2024-04-23T17:55:43.959 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T17:55:43.960 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --verbose --archive /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} --name leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/orig.config.yaml 2024-04-23T17:55:44.024 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T17:55:44.128 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: null first_in_suite: false job_id: '7670091' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5182 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLStkaM79fMI2aiYOFzfuOrR7Pq5GZtlS9Df+QqNdgyw0pmUlEUf7LwEw6yIuoHeD/tX4MYrJ09Pif9wFHUJgXQ= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMiGL3h/cBsdICJfyfAwFkBowhz33uEV0Dy+8E6s9MPLwoYkM4k/+/l4dMnrMkScPGYSGCrJ8By3qFroDIEzct4= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAkVgZ4naHz37gpAwJEFiEu8ngNYQc1NzoQpuVbbXr4+aVIvC6jXHEerQMAiU/Zn8grtVSSqYF3BCYdcNFJnlGQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_17:44:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T17:55:44.128 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa; will attempt to use it 2024-04-23T17:55:44.129 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks 2024-04-23T17:55:44.129 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T17:55:44.131 INFO:teuthology.task.internal:Checking packages... 2024-04-23T17:55:44.154 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-23T17:55:44.154 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T17:55:44.155 INFO:teuthology.packaging:ref: None 2024-04-23T17:55:44.155 INFO:teuthology.packaging:tag: None 2024-04-23T17:55:44.155 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-23T17:55:44.155 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T17:55:44.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-23T17:55:44.355 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-23T17:55:44.357 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T17:55:44.364 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T17:55:44.365 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T17:55:44.375 INFO:teuthology.task.internal:Saving configuration 2024-04-23T17:55:44.398 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T17:55:44.404 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T17:55:44.424 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.353418', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLStkaM79fMI2aiYOFzfuOrR7Pq5GZtlS9Df+QqNdgyw0pmUlEUf7LwEw6yIuoHeD/tX4MYrJ09Pif9wFHUJgXQ='} 2024-04-23T17:55:44.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.351380', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMiGL3h/cBsdICJfyfAwFkBowhz33uEV0Dy+8E6s9MPLwoYkM4k/+/l4dMnrMkScPGYSGCrJ8By3qFroDIEzct4='} 2024-04-23T17:55:44.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.355171', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAkVgZ4naHz37gpAwJEFiEu8ngNYQc1NzoQpuVbbXr4+aVIvC6jXHEerQMAiU/Zn8grtVSSqYF3BCYdcNFJnlGQ='} 2024-04-23T17:55:44.460 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T17:55:44.467 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-23T17:55:44.467 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-23T17:55:44.467 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-23T17:55:44.467 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T17:55:44.568 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f887d2900d0>, signals=[15]) 2024-04-23T17:55:44.568 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T17:55:44.575 INFO:teuthology.task.internal:Opening connections... 2024-04-23T17:55:44.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T17:55:44.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:55:44.659 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-23T17:55:44.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:55:44.735 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-04-23T17:55:44.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:55:44.809 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T17:55:44.817 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-23T17:55:44.836 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-23T17:55:44.836 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:55:44.894 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:55:44.895 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:55:44.895 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:55:44.895 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:55:44.895 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:55:44.895 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-23T17:55:44.917 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-23T17:55:44.936 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-23T17:55:44.936 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-23T17:55:44.994 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-23T17:55:44.994 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-23T17:55:44.994 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:55:44.995 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:55:44.996 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-23T17:55:45.017 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-04-23T17:55:45.036 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-04-23T17:55:45.036 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:55:45.094 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:55:45.095 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:55:45.095 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:55:45.095 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:55:45.095 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-04-23T17:55:45.159 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T17:55:45.170 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T17:55:45.176 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T17:55:45.176 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:55:45.179 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:55:45.181 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:55:45.198 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T17:55:45.205 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T17:55:45.205 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:55:45.236 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:55:45.238 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:55:45.266 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T17:55:45.334 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T17:55:45.349 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-23T17:55:45.349 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-23T17:55:45.349 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-23T17:55:45.350 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:55:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:55:45.367 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-23T17:55:45.425 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-437.el9.x86_64 2024-04-23T17:55:45.425 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-437.el9.x86_64 2024-04-23T17:55:45.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-04-23T17:55:56.167 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 907 kB/s | 8.0 MB 00:09 2024-04-23T17:56:18.191 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 931 kB/s | 19 MB 00:20 2024-04-23T17:56:23.275 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-04-23T17:56:24.876 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-23T17:56:26.313 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-23T17:56:31.510 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-23T17:56:33.409 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T17:56:33.409 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T17:56:33.503 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-23T17:56:33.504 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-23T17:56:33.505 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-23T17:56:33.620 DEBUG:teuthology.orchestra.run.smithi032:> echo no | sudo yum reinstall kernel || true 2024-04-23T17:56:34.141 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 05:56:31 PM UTC. 2024-04-23T17:56:34.265 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-23T17:56:34.279 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.279 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalling: 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 255 k 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 0 2024-04-23T17:56:34.280 INFO:teuthology.orchestra.run.smithi032.stderr:Operation aborted. 2024-04-23T17:56:34.341 INFO:teuthology.orchestra.run.smithi032.stdout:Is this ok [y/N]: 2024-04-23T17:56:34.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum reinstall -y kernel || true 2024-04-23T17:56:34.855 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 05:56:31 PM UTC. 2024-04-23T17:56:34.976 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-23T17:56:34.976 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalling: 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 255 k 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 0 2024-04-23T17:56:34.977 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-23T17:56:35.863 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-437.el9.x86_64.rpm 409 kB/s | 255 kB 00:00 2024-04-23T17:56:35.863 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-23T17:56:35.863 INFO:teuthology.orchestra.run.smithi032.stdout:Total 288 kB/s | 255 kB 00:00 2024-04-23T17:56:35.882 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-23T17:56:36.081 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-23T17:56:36.081 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-23T17:56:36.086 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-23T17:56:36.087 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-23T17:56:36.248 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-23T17:56:36.322 INFO:teuthology.orchestra.run.smithi032.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T17:56:36.322 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:56:36.956 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:56:36.957 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T17:56:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:56:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:37.676 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalled: 2024-04-23T17:56:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T17:56:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T17:56:37.676 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-23T17:56:37.878 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T17:56:37.926 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T17:56:37.926 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T17:56:37.927 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T17:56:37.927 DEBUG:teuthology.parallel:result is None 2024-04-23T17:56:37.927 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T17:56:37.935 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T17:56:37.935 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:56:37.938 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:56:37.940 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:56:37.959 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T17:56:37.965 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T17:56:37.976 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T17:56:37.976 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:56:37.994 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:56:37.999 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:56:38.033 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T17:56:38.040 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T17:56:38.040 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:56:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:56:38.081 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:56:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:56:38.098 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:56:38.101 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:56:38.124 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:56:38.144 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.155 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.162 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.171 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.175 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.205 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:56:38.207 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T17:56:38.216 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T17:56:38.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:56:38.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:56:38.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:56:38.275 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T17:56:38.284 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T17:56:38.284 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:56:38.286 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:56:38.288 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:56:38.331 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:38.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:38.439 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:38.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:38.560 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T17:56:38.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:56:38.624 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:38.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:38.722 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:38.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:38.848 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T17:56:38.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:56:38.915 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:38.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:56:39.009 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:39.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:56:39.137 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T17:56:39.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:56:39.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-23T17:56:39.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-23T17:56:39.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-04-23T17:56:39.242 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:56:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:56:39.286 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:56:39.711 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T17:56:39.723 INFO:teuthology.task.internal:Starting timer... 2024-04-23T17:56:39.723 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T17:56:39.733 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T17:56:39.744 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-23T17:56:39.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-23T17:56:39.791 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-23T17:56:39.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-23T17:56:39.846 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-23T17:56:39.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-04-23T17:56:39.913 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-04-23T17:56:39.915 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T17:56:39.915 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-23T17:56:39.940 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-23T17:56:39.940 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-23T17:56:39.979 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-23T17:56:39.979 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-04-23T17:56:40.000 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-04-23T17:56:40.001 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive'} 2024-04-23T17:56:40.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:56:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:56:40.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:56:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:56:40.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:56:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:56:40.083 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T17:56:40.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:56:40.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:56:40.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:56:40.180 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T17:56:40.190 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T17:56:40.195 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T17:56:40.329 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T17:56:40.344 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T17:56:40.346 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi064.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T18:00:45.903 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com')] 2024-04-23T18:00:45.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-23T18:00:45.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:45.982 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-23T18:00:46.062 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-23T18:00:46.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-23T18:00:46.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:46.142 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-23T18:00:46.221 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-23T18:00:46.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-23T18:00:46.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:46.315 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-04-23T18:00:46.397 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-23T18:00:46.397 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T18:00:46.411 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T18:00:46.411 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:00:46.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:00:46.414 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:00:46.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:00:46.417 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:00:46.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:00:46.444 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:00:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:00:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:00:46.466 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:00:46.480 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:00:46.483 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-23T18:00:46.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-23T18:00:46.494 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-23T18:00:46.498 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-23T18:00:46.504 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-04-23T18:00:46.508 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:00:46.511 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:00:46.515 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-04-23T18:00:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:00:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:00:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:00:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:00:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:00:46.578 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:00:46.578 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:00:46.578 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-23T18:00:46.579 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.579 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.579 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.579 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.581 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:00:46.581 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-23T18:00:46.581 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.582 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.582 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.582 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.606 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:00:46.609 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:00:46.609 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-23T18:00:46.609 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.610 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.610 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.610 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:00:46.611 INFO:teuthology.run_tasks:Running task install... 2024-04-23T18:00:46.620 DEBUG:teuthology.task.install:project ceph 2024-04-23T18:00:46.620 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-23T18:00:46.620 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-23T18:00:46.620 INFO:teuthology.task.install:Using flavor: default 2024-04-23T18:00:46.626 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T18:00:46.626 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T18:00:46.626 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:00:46.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:00:46.629 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:00:46.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:00:46.630 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:00:46.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:00:46.777 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:00:46.777 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:00:46.853 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:00:46.853 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:00:46.855 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:00:46.855 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:00:46.896 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:00:46.897 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:00:46.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:00:46.931 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:00:46.931 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:00:46.969 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:00:46.969 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:00:46.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:00:46.995 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:00:46.995 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:00:46.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:00:47.030 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:00:47.030 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:00:47.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:00:47.057 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:00:47.058 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:00:47.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:00:47.070 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:00:47.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:00:47.135 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:00:47.136 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-04-23T18:00:47.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-04-23T18:00:47.199 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-23T18:00:47.200 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:00:47.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-23T18:00:47.264 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-23T18:00:47.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-23T18:00:47.371 INFO:teuthology.orchestra.run.smithi064.stdout:48 files removed 2024-04-23T18:00:47.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:00:47.433 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-23T18:00:47.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:00:47.501 INFO:teuthology.orchestra.run.smithi032.stdout:48 files removed 2024-04-23T18:00:47.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:00:48.070 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-04-23T18:00:48.121 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 216 kB/s | 83 kB 00:00 2024-04-23T18:00:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 213 kB/s | 83 kB 00:00 2024-04-23T18:00:48.445 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-23T18:00:48.463 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 39 kB/s | 12 kB 00:00 2024-04-23T18:00:48.572 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-23T18:00:48.804 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-23T18:00:48.822 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-23T18:00:48.989 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-23T18:00:49.506 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-23T18:00:52.415 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-04-23T18:00:56.689 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-04-23T18:00:57.157 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-04-23T18:01:01.325 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-23T18:01:01.866 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.2 MB 00:01 2024-04-23T18:01:03.101 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-23T18:01:03.271 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 571 kB/s | 8.0 MB 00:14 2024-04-23T18:01:03.666 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-23T18:01:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-04-23T18:01:11.685 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-23T18:01:12.386 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 21 MB 00:08 2024-04-23T18:01:13.938 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:13.939 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:13.950 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:01:13.951 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:01:13.952 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:01:13.952 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:01:13.954 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:01:13.956 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:01:13.957 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:01:14.024 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:01:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:01:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:01:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:01:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:01:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:01:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:01:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:01:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:01:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:01:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:01:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:01:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:01:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-23T18:01:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:Install 148 Packages 2024-04-23T18:01:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 4 Packages 2024-04-23T18:01:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:01:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 222 M 2024-04-23T18:01:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-23T18:01:17.576 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-23T18:01:19.778 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:19.779 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:19.790 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:01:19.791 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:01:19.792 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:01:19.793 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:01:19.794 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:01:19.797 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:01:19.798 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:01:19.866 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:01:19.880 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:01:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:01:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:01:19.883 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:01:19.884 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:01:19.885 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:01:19.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:01:19.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:01:19.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:01:19.889 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:01:19.890 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:01:19.891 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:01:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-23T18:01:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-23T18:01:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:Install 148 Packages 2024-04-23T18:01:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 4 Packages 2024-04-23T18:01:19.892 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:01:19.893 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 222 M 2024-04-23T18:01:19.893 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-23T18:01:25.511 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 921 kB/s | 19 MB 00:21 2024-04-23T18:01:31.352 INFO:teuthology.orchestra.run.smithi040.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-23T18:01:31.369 INFO:teuthology.orchestra.run.smithi032.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-23T18:01:31.570 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.2 MB 00:02 2024-04-23T18:01:31.719 INFO:teuthology.orchestra.run.smithi040.stdout:(2/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 8.1 MB/s | 5.4 MB 00:00 2024-04-23T18:01:31.777 INFO:teuthology.orchestra.run.smithi040.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.4 MB/s | 145 kB 00:00 2024-04-23T18:01:32.011 INFO:teuthology.orchestra.run.smithi040.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 10 MB/s | 2.4 MB 00:00 2024-04-23T18:01:32.170 INFO:teuthology.orchestra.run.smithi040.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 9.5 MB/s | 1.5 MB 00:00 2024-04-23T18:01:32.228 INFO:teuthology.orchestra.run.smithi032.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.0 MB/s | 890 kB 00:00 2024-04-23T18:01:32.445 INFO:teuthology.orchestra.run.smithi032.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 668 kB/s | 145 kB 00:00 2024-04-23T18:01:32.545 INFO:teuthology.orchestra.run.smithi040.stdout:(6/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 13 MB/s | 4.7 MB 00:00 2024-04-23T18:01:32.578 INFO:teuthology.orchestra.run.smithi040.stdout:(7/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 726 kB/s | 890 kB 00:01 2024-04-23T18:01:33.237 INFO:teuthology.orchestra.run.smithi040.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 9.7 MB/s | 21 MB 00:02 2024-04-23T18:01:33.380 INFO:teuthology.orchestra.run.smithi040.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 178 kB/s | 25 kB 00:00 2024-04-23T18:01:33.829 INFO:teuthology.orchestra.run.smithi040.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 13 MB/s | 17 MB 00:01 2024-04-23T18:01:33.913 INFO:teuthology.orchestra.run.smithi040.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.6 MB/s | 225 kB 00:00 2024-04-23T18:01:33.980 INFO:teuthology.orchestra.run.smithi040.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 407 kB/s | 27 kB 00:00 2024-04-23T18:01:33.987 INFO:teuthology.orchestra.run.smithi032.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.5 MB/s | 2.4 MB 00:01 2024-04-23T18:01:34.038 INFO:teuthology.orchestra.run.smithi040.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 544 kB/s | 32 kB 00:00 2024-04-23T18:01:34.138 INFO:teuthology.orchestra.run.smithi040.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 7.3 MB/s | 747 kB 00:00 2024-04-23T18:01:34.214 INFO:teuthology.orchestra.run.smithi040.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.1 MB/s | 163 kB 00:00 2024-04-23T18:01:34.289 INFO:teuthology.orchestra.run.smithi040.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.6 MB/s | 126 kB 00:00 2024-04-23T18:01:34.389 INFO:teuthology.orchestra.run.smithi040.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 4.9 MB/s | 500 kB 00:00 2024-04-23T18:01:35.073 INFO:teuthology.orchestra.run.smithi032.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 1.4 MB/s | 1.5 MB 00:01 2024-04-23T18:01:35.074 INFO:teuthology.orchestra.run.smithi040.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 7.8 MB/s | 5.4 MB 00:00 2024-04-23T18:01:35.156 INFO:teuthology.orchestra.run.smithi040.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 542 kB/s | 45 kB 00:00 2024-04-23T18:01:35.240 INFO:teuthology.orchestra.run.smithi040.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.6 MB/s | 135 kB 00:00 2024-04-23T18:01:35.324 INFO:teuthology.orchestra.run.smithi040.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.9 MB/s | 161 kB 00:00 2024-04-23T18:01:35.415 INFO:teuthology.orchestra.run.smithi040.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.4 MB/s | 320 kB 00:00 2024-04-23T18:01:35.483 INFO:teuthology.orchestra.run.smithi040.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 4.4 MB/s | 298 kB 00:00 2024-04-23T18:01:35.541 INFO:teuthology.orchestra.run.smithi040.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 99 kB 00:00 2024-04-23T18:01:35.600 INFO:teuthology.orchestra.run.smithi040.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.4 MB/s | 85 kB 00:00 2024-04-23T18:01:35.629 INFO:teuthology.orchestra.run.smithi032.stdout:(6/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 4.8 MB/s | 21 MB 00:04 2024-04-23T18:01:36.021 INFO:teuthology.orchestra.run.smithi032.stdout:(7/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.1 MB/s | 5.4 MB 00:04 2024-04-23T18:01:36.092 INFO:teuthology.orchestra.run.smithi040.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 6.3 MB/s | 3.1 MB 00:00 2024-04-23T18:01:36.167 INFO:teuthology.orchestra.run.smithi040.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 170 kB 00:00 2024-04-23T18:01:36.242 INFO:teuthology.orchestra.run.smithi040.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 457 kB/s | 34 kB 00:00 2024-04-23T18:01:36.326 INFO:teuthology.orchestra.run.smithi040.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.7 MB/s | 142 kB 00:00 2024-04-23T18:01:37.002 INFO:teuthology.orchestra.run.smithi040.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 5.2 MB/s | 3.5 MB 00:00 2024-04-23T18:01:37.354 INFO:teuthology.orchestra.run.smithi032.stdout:(8/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 9.8 MB/s | 17 MB 00:01 2024-04-23T18:01:37.454 INFO:teuthology.orchestra.run.smithi032.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 252 kB/s | 25 kB 00:00 2024-04-23T18:01:37.796 INFO:teuthology.orchestra.run.smithi032.stdout:(10/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x8 1.7 MB/s | 4.7 MB 00:02 2024-04-23T18:01:37.905 INFO:teuthology.orchestra.run.smithi032.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.0 MB/s | 225 kB 00:00 2024-04-23T18:01:37.964 INFO:teuthology.orchestra.run.smithi032.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 461 kB/s | 27 kB 00:00 2024-04-23T18:01:38.014 INFO:teuthology.orchestra.run.smithi032.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 634 kB/s | 32 kB 00:00 2024-04-23T18:01:38.198 INFO:teuthology.orchestra.run.smithi032.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.0 MB/s | 747 kB 00:00 2024-04-23T18:01:38.244 INFO:teuthology.orchestra.run.smithi040.stdout:(31/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 10 MB/s | 49 MB 00:04 2024-04-23T18:01:38.282 INFO:teuthology.orchestra.run.smithi032.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.9 MB/s | 163 kB 00:00 2024-04-23T18:01:38.321 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 2.9 kB/s | 16 kB 00:05 2024-04-23T18:01:38.344 INFO:teuthology.orchestra.run.smithi040.stdout:(32/152): ceph-mgr-diskprediction-local-19.0.0- 5.5 MB/s | 7.4 MB 00:01 2024-04-23T18:01:38.348 INFO:teuthology.orchestra.run.smithi032.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.9 MB/s | 126 kB 00:00 2024-04-23T18:01:38.369 INFO:teuthology.orchestra.run.smithi040.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.9 MB/s | 249 kB 00:00 2024-04-23T18:01:38.394 INFO:teuthology.orchestra.run.smithi040.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 987 kB/s | 49 kB 00:00 2024-04-23T18:01:38.419 INFO:teuthology.orchestra.run.smithi040.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 328 kB/s | 16 kB 00:00 2024-04-23T18:01:38.478 INFO:teuthology.orchestra.run.smithi040.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 273 kB 00:00 2024-04-23T18:01:38.515 INFO:teuthology.orchestra.run.smithi032.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.9 MB/s | 500 kB 00:00 2024-04-23T18:01:38.545 INFO:teuthology.orchestra.run.smithi040.stdout:(37/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 6.0 MB/s | 762 kB 00:00 2024-04-23T18:01:38.570 INFO:teuthology.orchestra.run.smithi040.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 199 kB/s | 18 kB 00:00 2024-04-23T18:01:38.679 INFO:teuthology.orchestra.run.smithi040.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 80 kB 00:00 2024-04-23T18:01:38.754 INFO:teuthology.orchestra.run.smithi040.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-04-23T18:01:38.779 INFO:teuthology.orchestra.run.smithi040.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-23T18:01:38.821 INFO:teuthology.orchestra.run.smithi040.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-23T18:01:38.863 INFO:teuthology.orchestra.run.smithi040.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-23T18:01:38.889 INFO:teuthology.orchestra.run.smithi040.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-23T18:01:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-23T18:01:38.931 INFO:teuthology.orchestra.run.smithi040.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-23T18:01:38.956 INFO:teuthology.orchestra.run.smithi040.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-04-23T18:01:39.206 INFO:teuthology.orchestra.run.smithi040.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-04-23T18:01:39.265 INFO:teuthology.orchestra.run.smithi040.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 943 kB 00:00 2024-04-23T18:01:39.290 INFO:teuthology.orchestra.run.smithi040.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 107 kB 00:00 2024-04-23T18:01:39.324 INFO:teuthology.orchestra.run.smithi040.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-23T18:01:39.349 INFO:teuthology.orchestra.run.smithi040.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-23T18:01:39.383 INFO:teuthology.orchestra.run.smithi040.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-04-23T18:01:39.442 INFO:teuthology.orchestra.run.smithi040.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-23T18:01:39.508 INFO:teuthology.orchestra.run.smithi040.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-04-23T18:01:39.567 INFO:teuthology.orchestra.run.smithi040.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-23T18:01:39.607 INFO:teuthology.orchestra.run.smithi032.stdout:(18/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 3.0 MB/s | 11 MB 00:03 2024-04-23T18:01:39.625 INFO:teuthology.orchestra.run.smithi040.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-04-23T18:01:39.676 INFO:teuthology.orchestra.run.smithi040.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.2 MB/s | 3.0 MB 00:00 2024-04-23T18:01:39.682 INFO:teuthology.orchestra.run.smithi032.stdout:(19/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.6 MB/s | 5.4 MB 00:01 2024-04-23T18:01:39.718 INFO:teuthology.orchestra.run.smithi040.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-04-23T18:01:39.733 INFO:teuthology.orchestra.run.smithi032.stdout:(20/152): python3-ceph-argparse-19.0.0-3156.gd6 359 kB/s | 45 kB 00:00 2024-04-23T18:01:39.746 INFO:teuthology.orchestra.run.smithi040.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-23T18:01:39.758 INFO:teuthology.orchestra.run.smithi032.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 1.7 MB/s | 135 kB 00:00 2024-04-23T18:01:39.785 INFO:teuthology.orchestra.run.smithi040.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-04-23T18:01:39.791 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-23T18:01:39.811 INFO:teuthology.orchestra.run.smithi040.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-23T18:01:39.817 INFO:teuthology.orchestra.run.smithi032.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.9 MB/s | 161 kB 00:00 2024-04-23T18:01:39.842 INFO:teuthology.orchestra.run.smithi032.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.7 MB/s | 320 kB 00:00 2024-04-23T18:01:39.869 INFO:teuthology.orchestra.run.smithi040.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-23T18:01:39.895 INFO:teuthology.orchestra.run.smithi040.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-23T18:01:39.928 INFO:teuthology.orchestra.run.smithi040.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-23T18:01:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 99 kB 00:00 2024-04-23T18:01:40.101 INFO:teuthology.orchestra.run.smithi032.stdout:(25/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.0 MB/s | 298 kB 00:00 2024-04-23T18:01:40.127 INFO:teuthology.orchestra.run.smithi032.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 444 kB/s | 85 kB 00:00 2024-04-23T18:01:40.194 INFO:teuthology.orchestra.run.smithi032.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.5 MB/s | 170 kB 00:00 2024-04-23T18:01:40.195 INFO:teuthology.orchestra.run.smithi040.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-04-23T18:01:40.252 INFO:teuthology.orchestra.run.smithi032.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 589 kB/s | 34 kB 00:00 2024-04-23T18:01:40.262 INFO:teuthology.orchestra.run.smithi040.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-23T18:01:40.328 INFO:teuthology.orchestra.run.smithi032.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.9 MB/s | 142 kB 00:00 2024-04-23T18:01:40.337 INFO:teuthology.orchestra.run.smithi040.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-23T18:01:40.396 INFO:teuthology.orchestra.run.smithi040.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-23T18:01:40.421 INFO:teuthology.orchestra.run.smithi040.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-23T18:01:40.521 INFO:teuthology.orchestra.run.smithi040.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-23T18:01:40.588 INFO:teuthology.orchestra.run.smithi040.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-23T18:01:40.730 INFO:teuthology.orchestra.run.smithi040.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-23T18:01:40.778 INFO:teuthology.orchestra.run.smithi032.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 4.6 MB/s | 3.1 MB 00:00 2024-04-23T18:01:40.789 INFO:teuthology.orchestra.run.smithi040.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-23T18:01:40.848 INFO:teuthology.orchestra.run.smithi040.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-23T18:01:40.923 INFO:teuthology.orchestra.run.smithi040.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-04-23T18:01:40.948 INFO:teuthology.orchestra.run.smithi040.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 255 kB/s | 26 kB 00:00 2024-04-23T18:01:40.982 INFO:teuthology.orchestra.run.smithi040.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-04-23T18:01:41.007 INFO:teuthology.orchestra.run.smithi040.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-23T18:01:41.041 INFO:teuthology.orchestra.run.smithi040.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-23T18:01:41.116 INFO:teuthology.orchestra.run.smithi040.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-23T18:01:41.203 INFO:teuthology.orchestra.run.smithi032.stdout:(31/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 13 MB/s | 49 MB 00:03 2024-04-23T18:01:41.208 INFO:teuthology.orchestra.run.smithi040.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-23T18:01:41.267 INFO:teuthology.orchestra.run.smithi040.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T18:01:41.287 INFO:teuthology.orchestra.run.smithi032.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 3.7 MB/s | 3.5 MB 00:00 2024-04-23T18:01:41.320 INFO:teuthology.orchestra.run.smithi032.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.1 MB/s | 249 kB 00:00 2024-04-23T18:01:41.333 INFO:teuthology.orchestra.run.smithi040.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-23T18:01:41.345 INFO:teuthology.orchestra.run.smithi032.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 846 kB/s | 49 kB 00:00 2024-04-23T18:01:41.359 INFO:teuthology.orchestra.run.smithi040.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 459 kB/s | 42 kB 00:00 2024-04-23T18:01:41.371 INFO:teuthology.orchestra.run.smithi032.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 328 kB/s | 16 kB 00:00 2024-04-23T18:01:41.394 INFO:teuthology.orchestra.run.smithi040.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.3 MB/s | 269 kB 00:00 2024-04-23T18:01:41.421 INFO:teuthology.orchestra.run.smithi032.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.5 MB/s | 273 kB 00:00 2024-04-23T18:01:41.422 INFO:teuthology.orchestra.run.smithi040.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-23T18:01:41.454 INFO:teuthology.orchestra.run.smithi040.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-23T18:01:41.463 INFO:teuthology.orchestra.run.smithi032.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 438 kB/s | 18 kB 00:00 2024-04-23T18:01:41.479 INFO:teuthology.orchestra.run.smithi040.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 873 kB/s | 51 kB 00:00 2024-04-23T18:01:41.504 INFO:teuthology.orchestra.run.smithi040.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-23T18:01:41.505 INFO:teuthology.orchestra.run.smithi032.stdout:(38/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 5.6 MB/s | 762 kB 00:00 2024-04-23T18:01:41.554 INFO:teuthology.orchestra.run.smithi040.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 838 kB/s | 42 kB 00:00 2024-04-23T18:01:41.597 INFO:teuthology.orchestra.run.smithi032.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-23T18:01:41.613 INFO:teuthology.orchestra.run.smithi040.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 755 kB/s | 44 kB 00:00 2024-04-23T18:01:41.664 INFO:teuthology.orchestra.run.smithi032.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-23T18:01:41.672 INFO:teuthology.orchestra.run.smithi040.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-23T18:01:41.722 INFO:teuthology.orchestra.run.smithi040.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 723 kB/s | 36 kB 00:00 2024-04-23T18:01:41.722 INFO:teuthology.orchestra.run.smithi032.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-23T18:01:41.748 INFO:teuthology.orchestra.run.smithi032.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-23T18:01:41.772 INFO:teuthology.orchestra.run.smithi040.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-23T18:01:41.798 INFO:teuthology.orchestra.run.smithi032.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-23T18:01:41.823 INFO:teuthology.orchestra.run.smithi032.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-23T18:01:41.831 INFO:teuthology.orchestra.run.smithi040.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-23T18:01:41.848 INFO:teuthology.orchestra.run.smithi032.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-23T18:01:41.874 INFO:teuthology.orchestra.run.smithi032.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-23T18:01:41.899 INFO:teuthology.orchestra.run.smithi032.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-23T18:01:42.034 INFO:teuthology.orchestra.run.smithi040.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-23T18:01:42.048 INFO:teuthology.orchestra.run.smithi040.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 266 kB/s | 58 kB 00:00 2024-04-23T18:01:42.158 INFO:teuthology.orchestra.run.smithi032.stdout:(48/152): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-23T18:01:42.182 INFO:teuthology.orchestra.run.smithi040.stdout:(99/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 1.1 MB/s | 11 MB 00:09 2024-04-23T18:01:42.199 INFO:teuthology.orchestra.run.smithi040.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-04-23T18:01:42.200 INFO:teuthology.orchestra.run.smithi032.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-23T18:01:42.224 INFO:teuthology.orchestra.run.smithi040.stdout:(101/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 488 kB/s | 94 kB 00:00 2024-04-23T18:01:42.234 INFO:teuthology.orchestra.run.smithi032.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-04-23T18:01:42.259 INFO:teuthology.orchestra.run.smithi032.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 367 kB/s | 21 kB 00:00 2024-04-23T18:01:42.266 INFO:teuthology.orchestra.run.smithi040.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-04-23T18:01:42.293 INFO:teuthology.orchestra.run.smithi032.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-23T18:01:42.300 INFO:teuthology.orchestra.run.smithi040.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-23T18:01:42.318 INFO:teuthology.orchestra.run.smithi032.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-04-23T18:01:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:(104/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 297 kB/s | 45 kB 00:00 2024-04-23T18:01:42.351 INFO:teuthology.orchestra.run.smithi032.stdout:(54/152): boost-program-options-1.75.0-8.el9.x8 554 kB/s | 107 kB 00:00 2024-04-23T18:01:42.377 INFO:teuthology.orchestra.run.smithi032.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-23T18:01:42.410 INFO:teuthology.orchestra.run.smithi032.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-23T18:01:42.427 INFO:teuthology.orchestra.run.smithi032.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-23T18:01:42.469 INFO:teuthology.orchestra.run.smithi032.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 773 kB/s | 45 kB 00:00 2024-04-23T18:01:42.528 INFO:teuthology.orchestra.run.smithi032.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-23T18:01:42.553 INFO:teuthology.orchestra.run.smithi032.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-23T18:01:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-23T18:01:42.591 INFO:teuthology.orchestra.run.smithi040.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-23T18:01:42.612 INFO:teuthology.orchestra.run.smithi032.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-04-23T18:01:42.637 INFO:teuthology.orchestra.run.smithi032.stdout:(63/152): libtool-2.4.6-45.el9.x86_64.rpm 3.4 MB/s | 582 kB 00:00 2024-04-23T18:01:42.663 INFO:teuthology.orchestra.run.smithi032.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-23T18:01:42.688 INFO:teuthology.orchestra.run.smithi032.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-23T18:01:42.713 INFO:teuthology.orchestra.run.smithi032.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 466 kB/s | 35 kB 00:00 2024-04-23T18:01:42.717 INFO:teuthology.orchestra.run.smithi040.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 741 kB/s | 309 kB 00:00 2024-04-23T18:01:42.767 INFO:teuthology.orchestra.run.smithi040.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-23T18:01:42.792 INFO:teuthology.orchestra.run.smithi040.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-23T18:01:42.843 INFO:teuthology.orchestra.run.smithi040.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-04-23T18:01:42.855 INFO:teuthology.orchestra.run.smithi032.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-23T18:01:42.868 INFO:teuthology.orchestra.run.smithi040.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-04-23T18:01:42.927 INFO:teuthology.orchestra.run.smithi040.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-23T18:01:42.989 INFO:teuthology.orchestra.run.smithi032.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-23T18:01:43.022 INFO:teuthology.orchestra.run.smithi032.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 5.3 MB/s | 911 kB 00:00 2024-04-23T18:01:43.035 INFO:teuthology.orchestra.run.smithi040.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-23T18:01:43.048 INFO:teuthology.orchestra.run.smithi032.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-23T18:01:43.060 INFO:teuthology.orchestra.run.smithi040.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-23T18:01:43.106 INFO:teuthology.orchestra.run.smithi032.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-23T18:01:43.127 INFO:teuthology.orchestra.run.smithi040.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-04-23T18:01:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 370 kB/s | 43 kB 00:00 2024-04-23T18:01:43.165 INFO:teuthology.orchestra.run.smithi032.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-23T18:01:43.178 INFO:teuthology.orchestra.run.smithi040.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-04-23T18:01:43.190 INFO:teuthology.orchestra.run.smithi032.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-23T18:01:43.203 INFO:teuthology.orchestra.run.smithi040.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-04-23T18:01:43.249 INFO:teuthology.orchestra.run.smithi032.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-23T18:01:43.270 INFO:teuthology.orchestra.run.smithi040.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-23T18:01:43.296 INFO:teuthology.orchestra.run.smithi040.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-23T18:01:43.316 INFO:teuthology.orchestra.run.smithi032.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-04-23T18:01:43.321 INFO:teuthology.orchestra.run.smithi040.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-23T18:01:43.346 INFO:teuthology.orchestra.run.smithi040.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-04-23T18:01:43.388 INFO:teuthology.orchestra.run.smithi040.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-23T18:01:43.391 INFO:teuthology.orchestra.run.smithi032.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-04-23T18:01:43.418 INFO:teuthology.orchestra.run.smithi032.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 250 kB/s | 26 kB 00:00 2024-04-23T18:01:43.444 INFO:teuthology.orchestra.run.smithi032.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 366 kB/s | 19 kB 00:00 2024-04-23T18:01:43.446 INFO:teuthology.orchestra.run.smithi040.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-04-23T18:01:43.472 INFO:teuthology.orchestra.run.smithi040.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-23T18:01:43.497 INFO:teuthology.orchestra.run.smithi040.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 958 kB/s | 144 kB 00:00 2024-04-23T18:01:43.519 INFO:teuthology.orchestra.run.smithi032.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-23T18:01:43.522 INFO:teuthology.orchestra.run.smithi040.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-23T18:01:43.544 INFO:teuthology.orchestra.run.smithi032.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-04-23T18:01:43.548 INFO:teuthology.orchestra.run.smithi040.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-04-23T18:01:43.569 INFO:teuthology.orchestra.run.smithi032.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-23T18:01:43.573 INFO:teuthology.orchestra.run.smithi040.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-23T18:01:43.603 INFO:teuthology.orchestra.run.smithi032.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-23T18:01:43.615 INFO:teuthology.orchestra.run.smithi040.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-23T18:01:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-04-23T18:01:43.657 INFO:teuthology.orchestra.run.smithi040.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-04-23T18:01:43.662 INFO:teuthology.orchestra.run.smithi032.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T18:01:43.682 INFO:teuthology.orchestra.run.smithi040.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-23T18:01:43.695 INFO:teuthology.orchestra.run.smithi032.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 721 kB/s | 42 kB 00:00 2024-04-23T18:01:43.707 INFO:teuthology.orchestra.run.smithi040.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-04-23T18:01:43.721 INFO:teuthology.orchestra.run.smithi032.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-23T18:01:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-23T18:01:43.766 INFO:teuthology.orchestra.run.smithi040.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-23T18:01:43.783 INFO:teuthology.orchestra.run.smithi040.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-23T18:01:43.805 INFO:teuthology.orchestra.run.smithi032.stdout:(87/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-04-23T18:01:43.808 INFO:teuthology.orchestra.run.smithi040.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-23T18:01:43.830 INFO:teuthology.orchestra.run.smithi032.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-23T18:01:43.850 INFO:teuthology.orchestra.run.smithi040.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-23T18:01:43.855 INFO:teuthology.orchestra.run.smithi032.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-04-23T18:01:43.867 INFO:teuthology.orchestra.run.smithi040.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-04-23T18:01:43.882 INFO:teuthology.orchestra.run.smithi032.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-04-23T18:01:43.892 INFO:teuthology.orchestra.run.smithi040.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-23T18:01:43.914 INFO:teuthology.orchestra.run.smithi032.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 639 kB/s | 54 kB 00:00 2024-04-23T18:01:43.918 INFO:teuthology.orchestra.run.smithi040.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 880 kB/s | 59 kB 00:00 2024-04-23T18:01:43.939 INFO:teuthology.orchestra.run.smithi032.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-04-23T18:01:43.964 INFO:teuthology.orchestra.run.smithi032.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 879 kB/s | 44 kB 00:00 2024-04-23T18:01:43.998 INFO:teuthology.orchestra.run.smithi032.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-23T18:01:44.024 INFO:teuthology.orchestra.run.smithi032.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 612 kB/s | 36 kB 00:00 2024-04-23T18:01:44.037 INFO:teuthology.orchestra.run.smithi040.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 211 kB/s | 36 kB 00:00 2024-04-23T18:01:44.049 INFO:teuthology.orchestra.run.smithi032.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-23T18:01:44.077 INFO:teuthology.orchestra.run.smithi040.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 277 kB/s | 51 kB 00:00 2024-04-23T18:01:44.091 INFO:teuthology.orchestra.run.smithi032.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-23T18:01:44.102 INFO:teuthology.orchestra.run.smithi040.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-23T18:01:44.107 INFO:teuthology.orchestra.run.smithi032.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-04-23T18:01:44.127 INFO:teuthology.orchestra.run.smithi040.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 993 kB/s | 90 kB 00:00 2024-04-23T18:01:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-23T18:01:44.161 INFO:teuthology.orchestra.run.smithi040.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-23T18:01:44.166 INFO:teuthology.orchestra.run.smithi032.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T18:01:44.186 INFO:teuthology.orchestra.run.smithi040.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-23T18:01:44.200 INFO:teuthology.orchestra.run.smithi032.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-23T18:01:44.212 INFO:teuthology.orchestra.run.smithi040.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-23T18:01:44.242 INFO:teuthology.orchestra.run.smithi032.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-23T18:01:44.262 INFO:teuthology.orchestra.run.smithi040.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-23T18:01:44.421 INFO:teuthology.orchestra.run.smithi040.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-04-23T18:01:44.442 INFO:teuthology.orchestra.run.smithi032.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.7 MB/s | 548 kB 00:00 2024-04-23T18:01:44.501 INFO:teuthology.orchestra.run.smithi032.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 5.2 MB/s | 309 kB 00:00 2024-04-23T18:01:44.521 INFO:teuthology.orchestra.run.smithi040.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-23T18:01:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 8.2 MB/s | 3.1 MB 00:00 2024-04-23T18:01:44.668 INFO:teuthology.orchestra.run.smithi032.stdout:(105/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-23T18:01:44.702 INFO:teuthology.orchestra.run.smithi032.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-04-23T18:01:44.735 INFO:teuthology.orchestra.run.smithi032.stdout:(107/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-04-23T18:01:44.761 INFO:teuthology.orchestra.run.smithi032.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-04-23T18:01:44.803 INFO:teuthology.orchestra.run.smithi032.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-23T18:01:44.836 INFO:teuthology.orchestra.run.smithi032.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 483 kB/s | 49 kB 00:00 2024-04-23T18:01:44.862 INFO:teuthology.orchestra.run.smithi032.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-23T18:01:44.912 INFO:teuthology.orchestra.run.smithi032.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-04-23T18:01:44.962 INFO:teuthology.orchestra.run.smithi032.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 868 kB/s | 43 kB 00:00 2024-04-23T18:01:44.971 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-23T18:01:44.987 INFO:teuthology.orchestra.run.smithi032.stdout:(114/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-04-23T18:01:45.013 INFO:teuthology.orchestra.run.smithi032.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-04-23T18:01:45.147 INFO:teuthology.orchestra.run.smithi032.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 94 kB/s | 15 kB 00:00 2024-04-23T18:01:45.172 INFO:teuthology.orchestra.run.smithi032.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-04-23T18:01:45.197 INFO:teuthology.orchestra.run.smithi032.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-23T18:01:45.273 INFO:teuthology.orchestra.run.smithi032.stdout:(119/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-04-23T18:01:45.298 INFO:teuthology.orchestra.run.smithi032.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-23T18:01:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-23T18:01:45.348 INFO:teuthology.orchestra.run.smithi032.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-23T18:01:45.374 INFO:teuthology.orchestra.run.smithi032.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-23T18:01:45.399 INFO:teuthology.orchestra.run.smithi032.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-23T18:01:45.424 INFO:teuthology.orchestra.run.smithi032.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-23T18:01:45.450 INFO:teuthology.orchestra.run.smithi032.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-23T18:01:45.475 INFO:teuthology.orchestra.run.smithi032.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-23T18:01:45.496 INFO:teuthology.orchestra.run.smithi040.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 2.6 MB/s | 3.4 MB 00:01 2024-04-23T18:01:45.500 INFO:teuthology.orchestra.run.smithi032.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-23T18:01:45.502 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:01:45.502 INFO:teuthology.orchestra.run.smithi040.stdout:Total 8.7 MB/s | 222 MB 00:25 2024-04-23T18:01:45.542 INFO:teuthology.orchestra.run.smithi032.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.3 MB/s | 2.0 MB 00:00 2024-04-23T18:01:45.576 INFO:teuthology.orchestra.run.smithi032.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 618 kB/s | 46 kB 00:00 2024-04-23T18:01:45.601 INFO:teuthology.orchestra.run.smithi032.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-23T18:01:45.627 INFO:teuthology.orchestra.run.smithi032.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-23T18:01:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-23T18:01:45.677 INFO:teuthology.orchestra.run.smithi032.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-23T18:01:45.702 INFO:teuthology.orchestra.run.smithi032.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-23T18:01:45.719 INFO:teuthology.orchestra.run.smithi032.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 677 kB/s | 28 kB 00:00 2024-04-23T18:01:45.745 INFO:teuthology.orchestra.run.smithi032.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-23T18:01:45.761 INFO:teuthology.orchestra.run.smithi032.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-04-23T18:01:45.787 INFO:teuthology.orchestra.run.smithi032.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-04-23T18:01:45.812 INFO:teuthology.orchestra.run.smithi032.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-23T18:01:45.837 INFO:teuthology.orchestra.run.smithi032.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 640 kB/s | 59 kB 00:00 2024-04-23T18:01:45.863 INFO:teuthology.orchestra.run.smithi032.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-23T18:01:45.880 INFO:teuthology.orchestra.run.smithi032.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-23T18:01:45.905 INFO:teuthology.orchestra.run.smithi032.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-23T18:01:45.930 INFO:teuthology.orchestra.run.smithi032.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-04-23T18:01:45.955 INFO:teuthology.orchestra.run.smithi032.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-23T18:01:45.981 INFO:teuthology.orchestra.run.smithi032.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-04-23T18:01:46.084 INFO:teuthology.orchestra.run.smithi032.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-23T18:01:46.456 INFO:teuthology.orchestra.run.smithi032.stdout:(149/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 6.6 MB/s | 3.1 MB 00:00 2024-04-23T18:01:46.515 INFO:teuthology.orchestra.run.smithi032.stdout:(150/152): librados2-19.0.0-3156.gd69f13d8.el7. 6.1 MB/s | 3.4 MB 00:00 2024-04-23T18:01:46.594 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-23T18:01:46.615 INFO:teuthology.orchestra.run.smithi032.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.5 MB/s | 1.3 MB 00:00 2024-04-23T18:01:46.740 INFO:teuthology.orchestra.run.smithi032.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.4 MB/s | 2.4 MB 00:00 2024-04-23T18:01:46.743 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:01:46.743 INFO:teuthology.orchestra.run.smithi032.stdout:Total 6.8 MB/s | 222 MB 00:32 2024-04-23T18:01:46.753 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-23T18:01:46.753 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-23T18:01:47.157 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:47.158 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:01:47.169 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:01:47.170 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:01:47.171 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:01:47.172 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:01:47.175 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:01:47.176 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:01:47.240 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:01:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:01:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:01:47.263 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:01:47.264 INFO:teuthology.orchestra.run.smithi064.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:01:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:01:47.266 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:01:47.267 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:01:47.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:01:47.269 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:01:47.270 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:01:47.271 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:01:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-23T18:01:47.273 INFO:teuthology.orchestra.run.smithi064.stdout:Install 148 Packages 2024-04-23T18:01:47.274 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 4 Packages 2024-04-23T18:01:47.274 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:01:47.274 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 222 M 2024-04-23T18:01:47.274 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-23T18:01:47.837 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-23T18:01:47.994 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-23T18:01:47.994 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-23T18:01:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-23T18:01:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-23T18:01:49.870 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-23T18:01:49.871 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-23T18:01:54.953 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-23T18:01:55.042 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:01:55.089 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:01:55.144 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:01:55.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:01:55.210 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:01:55.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:01:55.606 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:01:55.711 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:01:55.818 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:01:55.845 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:01:55.929 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:01:55.966 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:01:56.004 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-23T18:01:56.111 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:01:56.140 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:01:56.150 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:01:56.181 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:01:56.205 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:01:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:01:56.238 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:01:56.263 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:01:56.307 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:01:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:01:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:01:56.430 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:01:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:01:56.473 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:01:56.644 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:01:56.661 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:01:56.676 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:01:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:01:56.788 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:01:56.884 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:01:56.896 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:01:56.923 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:01:56.925 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:01:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:01:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:01:57.037 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:01:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:01:57.085 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:01:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:01:57.160 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:01:57.190 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:01:57.200 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:01:57.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:01:57.260 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:01:57.286 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:01:57.312 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:01:57.341 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:01:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:01:57.391 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:01:57.421 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:01:57.456 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:01:57.483 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:01:57.487 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:01:57.505 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:01:57.527 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:01:57.531 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:01:57.559 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:01:57.631 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:01:57.674 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:01:57.715 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:01:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:01:57.752 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:01:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:01:57.896 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:01:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:01:57.961 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:01:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:01:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:01:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:01:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:01:58.182 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:01:58.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:01:58.242 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:01:58.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:01:58.365 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:01:58.394 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:01:58.430 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:01:58.508 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:01:58.513 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:01:58.541 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:01:58.584 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:01:58.591 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:01:58.612 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:01:58.626 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:01:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:01:58.687 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:01:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:01:58.736 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:01:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:01:58.815 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:01:58.914 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:01:58.966 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:01:58.977 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:01:59.032 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:01:59.033 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:01:59.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:01:59.152 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:01:59.189 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:01:59.237 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:01:59.280 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:01:59.323 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:01:59.366 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:01:59.509 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:01:59.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:01:59.583 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:01:59.584 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:01:59.626 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:01:59.670 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:01:59.705 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:01:59.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:01:59.748 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:01:59.759 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:01:59.786 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:01:59.791 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:01:59.816 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:01:59.828 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:01:59.841 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:01:59.919 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:01:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:02:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:02:00.048 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:02:00.090 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:02:00.095 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:02:00.118 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:02:00.152 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:02:00.177 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:02:00.192 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:02:00.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:02:00.223 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:02:00.251 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:02:00.253 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:02:00.279 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:02:00.298 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:02:00.304 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:02:00.334 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:02:00.373 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:02:00.377 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:02:00.405 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:02:00.428 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:02:00.496 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:02:00.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:02:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:02:00.575 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:02:00.623 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:02:00.624 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:02:00.655 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:02:00.697 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:02:00.743 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:02:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:02:00.803 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:02:00.829 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:02:00.870 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:02:00.912 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:02:00.936 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:02:01.014 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:02:01.144 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:02:01.186 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:02:01.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:02:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:02:01.288 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:02:01.327 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:02:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:02:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:02:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:02:01.467 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:02:01.501 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:02:01.566 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:02:01.591 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:02:01.607 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:02:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:02:01.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:02:01.671 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:02:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:02:01.743 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:02:01.808 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:02:01.829 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:02:01.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:02:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:02:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:02:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:02:02.149 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:02:02.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:02:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:02:02.711 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:02:02.745 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:02:02.850 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:02:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:02:02.907 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:02:02.996 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:02:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:02:03.031 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:02:03.205 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:02:03.236 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:02:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:02:03.997 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:02:04.123 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:02:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:02:04.394 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:02:04.452 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:02:04.481 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:02:04.810 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:02:04.852 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:04.926 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:04.957 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:02:04.993 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:02:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:02:05.498 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:02:05.508 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:02:05.529 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:05.536 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:02:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:02:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:02:05.898 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:06.031 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:02:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:02:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:02:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:06.698 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:06.719 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:02:08.038 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:08.078 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:08.649 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:08.689 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:08.782 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:08.782 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:02:08.782 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:02:08.782 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:08.816 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:02:09.097 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:09.133 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:09.704 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:09.838 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:09.838 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:02:09.838 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:02:09.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:02:10.229 INFO:teuthology.orchestra.run.smithi064.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-23T18:02:11.255 INFO:teuthology.orchestra.run.smithi064.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 868 kB/s | 890 kB 00:01 2024-04-23T18:02:11.480 INFO:teuthology.orchestra.run.smithi064.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 644 kB/s | 145 kB 00:00 2024-04-23T18:02:11.647 INFO:teuthology.orchestra.run.smithi064.stdout:(4/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 3.1 MB/s | 5.4 MB 00:01 2024-04-23T18:02:12.122 INFO:teuthology.orchestra.run.smithi064.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 3.2 MB/s | 1.5 MB 00:00 2024-04-23T18:02:12.839 INFO:teuthology.orchestra.run.smithi064.stdout:(6/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 2.4 MB 00:01 2024-04-23T18:02:13.364 INFO:teuthology.orchestra.run.smithi064.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 3.8 MB/s | 4.7 MB 00:01 2024-04-23T18:02:15.756 INFO:teuthology.orchestra.run.smithi064.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 4.4 MB/s | 11 MB 00:02 2024-04-23T18:02:15.822 INFO:teuthology.orchestra.run.smithi064.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 379 kB/s | 25 kB 00:00 2024-04-23T18:02:16.165 INFO:teuthology.orchestra.run.smithi064.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 3.4 MB/s | 21 MB 00:06 2024-04-23T18:02:16.298 INFO:teuthology.orchestra.run.smithi064.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 225 kB 00:00 2024-04-23T18:02:16.390 INFO:teuthology.orchestra.run.smithi064.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-23T18:02:16.457 INFO:teuthology.orchestra.run.smithi064.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 476 kB/s | 32 kB 00:00 2024-04-23T18:02:16.757 INFO:teuthology.orchestra.run.smithi064.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 2.4 MB/s | 747 kB 00:00 2024-04-23T18:02:16.849 INFO:teuthology.orchestra.run.smithi064.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 163 kB 00:00 2024-04-23T18:02:16.933 INFO:teuthology.orchestra.run.smithi064.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-23T18:02:17.117 INFO:teuthology.orchestra.run.smithi064.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.7 MB/s | 500 kB 00:00 2024-04-23T18:02:17.859 INFO:teuthology.orchestra.run.smithi064.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.4 MB/s | 17 MB 00:05 2024-04-23T18:02:17.959 INFO:teuthology.orchestra.run.smithi064.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 453 kB/s | 45 kB 00:00 2024-04-23T18:02:18.059 INFO:teuthology.orchestra.run.smithi064.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.3 MB/s | 135 kB 00:00 2024-04-23T18:02:18.176 INFO:teuthology.orchestra.run.smithi064.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.4 MB/s | 161 kB 00:00 2024-04-23T18:02:18.376 INFO:teuthology.orchestra.run.smithi064.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 320 kB 00:00 2024-04-23T18:02:18.485 INFO:teuthology.orchestra.run.smithi064.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.7 MB/s | 298 kB 00:00 2024-04-23T18:02:18.594 INFO:teuthology.orchestra.run.smithi064.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 920 kB/s | 99 kB 00:00 2024-04-23T18:02:18.719 INFO:teuthology.orchestra.run.smithi064.stdout:(25/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 3.3 MB/s | 5.4 MB 00:01 2024-04-23T18:02:18.744 INFO:teuthology.orchestra.run.smithi064.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 568 kB/s | 85 kB 00:00 2024-04-23T18:02:18.869 INFO:teuthology.orchestra.run.smithi064.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.3 MB/s | 170 kB 00:00 2024-04-23T18:02:18.995 INFO:teuthology.orchestra.run.smithi064.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 274 kB/s | 34 kB 00:00 2024-04-23T18:02:19.128 INFO:teuthology.orchestra.run.smithi064.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.0 MB/s | 142 kB 00:00 2024-04-23T18:02:19.978 INFO:teuthology.orchestra.run.smithi064.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.5 MB/s | 3.1 MB 00:01 2024-04-23T18:02:20.854 INFO:teuthology.orchestra.run.smithi064.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 2.1 MB/s | 3.5 MB 00:01 2024-04-23T18:02:20.990 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:02:20.990 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-23T18:02:20.990 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-23T18:02:20.991 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:21.154 INFO:teuthology.orchestra.run.smithi064.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 830 kB/s | 249 kB 00:00 2024-04-23T18:02:21.180 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:02:21.196 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:02:21.196 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:21.196 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:02:21.196 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:21.254 INFO:teuthology.orchestra.run.smithi064.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 493 kB/s | 49 kB 00:00 2024-04-23T18:02:21.354 INFO:teuthology.orchestra.run.smithi064.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 165 kB/s | 16 kB 00:00 2024-04-23T18:02:21.621 INFO:teuthology.orchestra.run.smithi064.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.0 MB/s | 273 kB 00:00 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-23T18:02:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:21.853 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:02:21.869 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:02:21.869 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:21.869 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:02:21.869 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:22.222 INFO:teuthology.orchestra.run.smithi064.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 1.2 MB/s | 762 kB 00:00 2024-04-23T18:02:22.314 INFO:teuthology.orchestra.run.smithi064.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 200 kB/s | 18 kB 00:00 2024-04-23T18:02:22.439 INFO:teuthology.orchestra.run.smithi064.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 643 kB/s | 80 kB 00:00 2024-04-23T18:02:22.556 INFO:teuthology.orchestra.run.smithi064.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-04-23T18:02:22.606 INFO:teuthology.orchestra.run.smithi064.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-23T18:02:22.665 INFO:teuthology.orchestra.run.smithi064.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.3 MB/s | 253 kB 00:00 2024-04-23T18:02:22.732 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:02:22.758 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:02:22.758 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:22.758 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:02:22.758 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:02:22.758 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:02:22.759 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:22.806 INFO:teuthology.orchestra.run.smithi064.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 8.7 MB/s | 1.2 MB 00:00 2024-04-23T18:02:22.857 INFO:teuthology.orchestra.run.smithi064.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-23T18:02:22.907 INFO:teuthology.orchestra.run.smithi064.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-04-23T18:02:22.958 INFO:teuthology.orchestra.run.smithi064.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-23T18:02:22.991 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:02:23.014 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:02:23.015 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:23.015 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:02:23.015 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:02:23.015 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:02:23.015 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:23.016 INFO:teuthology.orchestra.run.smithi064.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-23T18:02:23.414 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:02:23.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:23.450 INFO:teuthology.orchestra.run.smithi064.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-04-23T18:02:23.483 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:02:23.507 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:23.608 INFO:teuthology.orchestra.run.smithi064.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 5.8 MB/s | 943 kB 00:00 2024-04-23T18:02:23.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:02:23.672 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:02:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:02:23.684 INFO:teuthology.orchestra.run.smithi064.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:02:23.705 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:23.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:02:23.742 INFO:teuthology.orchestra.run.smithi064.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-23T18:02:23.834 INFO:teuthology.orchestra.run.smithi064.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-23T18:02:23.884 INFO:teuthology.orchestra.run.smithi064.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-04-23T18:02:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:02:23.985 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:02:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:02:24.110 INFO:teuthology.orchestra.run.smithi064.stdout:(53/152): ceph-mgr-diskprediction-local-19.0.0- 1.8 MB/s | 7.4 MB 00:04 2024-04-23T18:02:24.152 INFO:teuthology.orchestra.run.smithi064.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-04-23T18:02:24.155 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:02:24.174 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:02:24.197 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:02:24.197 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:24.198 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:02:24.198 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:02:24.198 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:02:24.198 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:24.219 INFO:teuthology.orchestra.run.smithi064.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.0 MB/s | 3.0 MB 00:00 2024-04-23T18:02:24.244 INFO:teuthology.orchestra.run.smithi064.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-23T18:02:24.297 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:02:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:02:24.343 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:02:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:02:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:02:24.394 INFO:teuthology.orchestra.run.smithi064.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 915 kB/s | 160 kB 00:00 2024-04-23T18:02:24.402 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:02:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-04-23T18:02:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:02:24.511 INFO:teuthology.orchestra.run.smithi064.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-23T18:02:24.545 INFO:teuthology.orchestra.run.smithi064.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-04-23T18:02:24.604 INFO:teuthology.orchestra.run.smithi064.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-04-23T18:02:24.621 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:02:24.629 INFO:teuthology.orchestra.run.smithi064.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-04-23T18:02:24.675 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:02:24.704 INFO:teuthology.orchestra.run.smithi064.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-23T18:02:24.754 INFO:teuthology.orchestra.run.smithi064.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-04-23T18:02:24.755 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:02:24.780 INFO:teuthology.orchestra.run.smithi064.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-23T18:02:24.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:02:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:02:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:02:24.951 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:02:24.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:02:25.014 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:02:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-23T18:02:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:02:25.049 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:02:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:02:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:02:25.305 INFO:teuthology.orchestra.run.smithi064.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-23T18:02:25.414 INFO:teuthology.orchestra.run.smithi064.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-04-23T18:02:25.514 INFO:teuthology.orchestra.run.smithi064.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-23T18:02:25.547 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:02:25.564 INFO:teuthology.orchestra.run.smithi064.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 911 kB 00:00 2024-04-23T18:02:25.567 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:02:25.599 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:02:25.664 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:02:25.690 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:02:25.756 INFO:teuthology.orchestra.run.smithi064.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-23T18:02:25.856 INFO:teuthology.orchestra.run.smithi064.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 877 kB/s | 87 kB 00:00 2024-04-23T18:02:26.147 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:02:26.184 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:02:26.382 INFO:teuthology.orchestra.run.smithi064.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 6.7 MB/s | 5.8 MB 00:00 2024-04-23T18:02:26.482 INFO:teuthology.orchestra.run.smithi064.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-23T18:02:26.582 INFO:teuthology.orchestra.run.smithi064.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 8.2 MB/s | 6.0 MB 00:00 2024-04-23T18:02:26.641 INFO:teuthology.orchestra.run.smithi064.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-23T18:02:26.666 INFO:teuthology.orchestra.run.smithi064.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 307 kB/s | 26 kB 00:00 2024-04-23T18:02:26.733 INFO:teuthology.orchestra.run.smithi064.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-04-23T18:02:26.783 INFO:teuthology.orchestra.run.smithi064.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-23T18:02:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:02:26.809 INFO:teuthology.orchestra.run.smithi064.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-04-23T18:02:26.833 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:02:26.901 INFO:teuthology.orchestra.run.smithi064.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-23T18:02:27.001 INFO:teuthology.orchestra.run.smithi064.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-23T18:02:27.093 INFO:teuthology.orchestra.run.smithi064.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 845 kB/s | 77 kB 00:00 2024-04-23T18:02:27.168 INFO:teuthology.orchestra.run.smithi064.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 564 kB/s | 42 kB 00:00 2024-04-23T18:02:27.200 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:02:27.226 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:02:27.277 INFO:teuthology.orchestra.run.smithi064.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-23T18:02:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:02:27.394 INFO:teuthology.orchestra.run.smithi064.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-23T18:02:27.506 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:02:27.537 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:02:27.561 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:27.612 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:02:27.623 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:02:27.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:02:27.779 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:02:27.862 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:02:27.869 INFO:teuthology.orchestra.run.smithi064.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 587 kB/s | 279 kB 00:00 2024-04-23T18:02:27.892 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:02:27.928 INFO:teuthology.orchestra.run.smithi064.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 877 kB/s | 51 kB 00:00 2024-04-23T18:02:28.000 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:02:28.003 INFO:teuthology.orchestra.run.smithi064.stdout:(89/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-04-23T18:02:28.028 INFO:teuthology.orchestra.run.smithi064.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 536 kB/s | 54 kB 00:00 2024-04-23T18:02:28.087 INFO:teuthology.orchestra.run.smithi064.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-04-23T18:02:28.173 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:02:28.179 INFO:teuthology.orchestra.run.smithi064.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 483 kB/s | 44 kB 00:00 2024-04-23T18:02:28.203 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:02:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:02:28.227 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:28.227 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:02:28.228 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:02:28.228 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:02:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:28.271 INFO:teuthology.orchestra.run.smithi064.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-04-23T18:02:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:02:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:02:28.329 INFO:teuthology.orchestra.run.smithi064.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-04-23T18:02:28.392 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:02:28.445 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:02:28.454 INFO:teuthology.orchestra.run.smithi064.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 380 kB/s | 47 kB 00:00 2024-04-23T18:02:28.546 INFO:teuthology.orchestra.run.smithi064.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.3 MB/s | 305 kB 00:00 2024-04-23T18:02:28.605 INFO:teuthology.orchestra.run.smithi064.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 993 kB/s | 58 kB 00:00 2024-04-23T18:02:28.680 INFO:teuthology.orchestra.run.smithi064.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-04-23T18:02:28.748 INFO:teuthology.orchestra.run.smithi064.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 957 kB/s | 64 kB 00:00 2024-04-23T18:02:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:02:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:02:28.805 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:02:28.805 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:28.805 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:02:28.805 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:02:28.805 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:02:28.806 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:28.849 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:02:28.871 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:02:28.871 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:28.872 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:02:28.872 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:28.916 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:02:28.939 INFO:teuthology.orchestra.run.smithi064.stdout:(100/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:02:28.940 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:28.956 INFO:teuthology.orchestra.run.smithi064.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 215 kB/s | 45 kB 00:00 2024-04-23T18:02:29.132 INFO:teuthology.orchestra.run.smithi064.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-04-23T18:02:29.157 INFO:teuthology.orchestra.run.smithi064.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-04-23T18:02:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:02:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:29.426 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:02:29.456 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:02:29.480 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:29.532 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:02:29.555 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:02:29.555 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:29.555 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:02:29.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:29.607 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:02:29.609 INFO:teuthology.orchestra.run.smithi064.stdout:(104/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 3.6 MB/s | 49 MB 00:13 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:02:29.631 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:29.859 INFO:teuthology.orchestra.run.smithi064.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-04-23T18:02:29.958 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:02:29.982 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:30.085 INFO:teuthology.orchestra.run.smithi064.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 333 kB/s | 309 kB 00:00 2024-04-23T18:02:30.151 INFO:teuthology.orchestra.run.smithi064.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 538 kB/s | 548 kB 00:01 2024-04-23T18:02:30.185 INFO:teuthology.orchestra.run.smithi064.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-23T18:02:30.252 INFO:teuthology.orchestra.run.smithi064.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-04-23T18:02:30.294 INFO:teuthology.orchestra.run.smithi064.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-04-23T18:02:30.361 INFO:teuthology.orchestra.run.smithi064.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-23T18:02:30.519 INFO:teuthology.orchestra.run.smithi064.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-23T18:02:30.544 INFO:teuthology.orchestra.run.smithi064.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-04-23T18:02:30.612 INFO:teuthology.orchestra.run.smithi064.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-04-23T18:02:30.637 INFO:teuthology.orchestra.run.smithi064.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 370 kB/s | 43 kB 00:00 2024-04-23T18:02:30.662 INFO:teuthology.orchestra.run.smithi064.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-23T18:02:30.712 INFO:teuthology.orchestra.run.smithi064.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-23T18:02:30.746 INFO:teuthology.orchestra.run.smithi064.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-23T18:02:30.779 INFO:teuthology.orchestra.run.smithi064.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-23T18:02:30.813 INFO:teuthology.orchestra.run.smithi064.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-23T18:02:30.846 INFO:teuthology.orchestra.run.smithi064.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-23T18:02:30.889 INFO:teuthology.orchestra.run.smithi064.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-04-23T18:02:30.955 INFO:teuthology.orchestra.run.smithi064.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-23T18:02:30.981 INFO:teuthology.orchestra.run.smithi064.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-23T18:02:31.006 INFO:teuthology.orchestra.run.smithi064.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-04-23T18:02:31.056 INFO:teuthology.orchestra.run.smithi064.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-23T18:02:31.082 INFO:teuthology.orchestra.run.smithi064.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-23T18:02:31.107 INFO:teuthology.orchestra.run.smithi064.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-23T18:02:31.157 INFO:teuthology.orchestra.run.smithi064.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-23T18:02:31.207 INFO:teuthology.orchestra.run.smithi064.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-23T18:02:31.258 INFO:teuthology.orchestra.run.smithi064.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-04-23T18:02:31.291 INFO:teuthology.orchestra.run.smithi064.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-04-23T18:02:31.316 INFO:teuthology.orchestra.run.smithi064.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-04-23T18:02:31.358 INFO:teuthology.orchestra.run.smithi064.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-23T18:02:31.383 INFO:teuthology.orchestra.run.smithi064.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-23T18:02:31.418 INFO:teuthology.orchestra.run.smithi064.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-04-23T18:02:31.459 INFO:teuthology.orchestra.run.smithi064.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-04-23T18:02:31.476 INFO:teuthology.orchestra.run.smithi064.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-04-23T18:02:31.518 INFO:teuthology.orchestra.run.smithi064.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-23T18:02:31.552 INFO:teuthology.orchestra.run.smithi064.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-04-23T18:02:31.568 INFO:teuthology.orchestra.run.smithi064.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-04-23T18:02:31.610 INFO:teuthology.orchestra.run.smithi064.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-23T18:02:31.653 INFO:teuthology.orchestra.run.smithi064.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-23T18:02:31.669 INFO:teuthology.orchestra.run.smithi064.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-04-23T18:02:31.745 INFO:teuthology.orchestra.run.smithi064.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-23T18:02:31.762 INFO:teuthology.orchestra.run.smithi064.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-23T18:02:31.787 INFO:teuthology.orchestra.run.smithi064.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-23T18:02:31.945 INFO:teuthology.orchestra.run.smithi064.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-23T18:02:32.196 INFO:teuthology.orchestra.run.smithi064.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 1.3 MB 00:00 2024-04-23T18:02:32.446 INFO:teuthology.orchestra.run.smithi064.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 2.4 MB 00:00 2024-04-23T18:02:34.146 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:02:34.188 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:02:34.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:02:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:02:34.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:02:34.313 INFO:teuthology.orchestra.run.smithi064.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 3.1 MB 00:02 2024-04-23T18:02:34.472 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:02:34.517 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:02:34.547 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:02:34.721 INFO:teuthology.orchestra.run.smithi064.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 1.1 MB/s | 3.4 MB 00:02 2024-04-23T18:02:34.723 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:02:34.724 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.7 MB/s | 222 MB 00:47 2024-04-23T18:02:34.871 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:02:34.912 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:02:34.954 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:02:34.985 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:02:35.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:02:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:35.217 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:02:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:35.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:02:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:35.289 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:02:35.846 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-23T18:02:35.865 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:35.903 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:35.924 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:02:36.005 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-23T18:02:36.005 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-23T18:02:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:02:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:02:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:02:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:02:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:02:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:02:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:02:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:02:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:02:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:02:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:36.848 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:02:36.934 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:02:36.968 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:02:37.062 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:02:37.097 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:02:37.097 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:02:37.189 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:02:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:02:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:02:37.312 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:02:37.382 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:37.878 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-23T18:02:37.878 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:02:42.292 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:02:42.293 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:02:42.294 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:02:42.294 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:02:42.294 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:02:42.294 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:02:42.294 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:02:42.306 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:02:42.306 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:02:42.306 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:02:42.306 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:02:42.306 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:02:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:02:42.308 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:02:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:02:42.310 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:02:42.311 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:02:42.312 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:02:42.313 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:02:42.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:02:42.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:02:42.316 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:02:43.145 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:02:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:02:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:02:43.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:02:43.160 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:02:43.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:02:43.162 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:02:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:02:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:02:43.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:02:43.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:02:43.167 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:02:43.168 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:02:43.902 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-23T18:02:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:02:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:02:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:02:44.134 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:02:44.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:02:44.211 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:02:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:44.576 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.578 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:02:44.579 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.580 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:44.581 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:02:44.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:02:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:02:44.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:02:44.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:02:44.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:02:44.587 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:02:44.588 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:02:44.589 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:02:44.589 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-23T18:02:44.589 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-23T18:02:44.660 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:02:44.785 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:02:44.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:02:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:02:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:02:44.965 DEBUG:teuthology.parallel:result is None 2024-04-23T18:02:45.083 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:02:45.138 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:02:45.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:45.216 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.218 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:02:45.219 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.220 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:02:45.221 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:02:45.222 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:02:45.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:02:45.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:02:45.225 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:02:45.226 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:02:45.227 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:02:45.228 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:02:45.229 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:02:45.230 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:02:45.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:02:45.230 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-23T18:02:45.254 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:02:45.291 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:02:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:02:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:02:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:02:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:02:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:02:45.630 DEBUG:teuthology.parallel:result is None 2024-04-23T18:02:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:02:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:02:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:02:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:02:45.961 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:02:46.000 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:02:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:02:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:02:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:02:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:02:46.267 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:02:46.299 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:02:46.343 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:02:46.432 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:02:46.461 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:02:46.505 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:02:46.533 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:02:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:02:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:02:46.736 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:02:46.880 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:02:46.936 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:02:47.498 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:02:47.582 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:02:47.616 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:02:47.660 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:02:47.697 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:02:47.740 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:02:47.902 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:02:47.967 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:02:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:02:48.099 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:02:48.134 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:02:48.170 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:02:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:02:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:02:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:02:48.339 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:02:48.485 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:02:48.527 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:02:48.557 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:02:48.600 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:02:48.699 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:02:48.732 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:02:48.810 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:02:49.049 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:02:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:02:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:02:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:02:49.329 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:02:49.357 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:02:49.392 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:02:49.431 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:02:49.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:02:49.500 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:02:49.535 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:02:49.568 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:02:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:02:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:02:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:02:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:02:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:02:49.854 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:02:49.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:02:50.017 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:02:50.801 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:02:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:02:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:02:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:02:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:02:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:02:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:02:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:02:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:02:51.468 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:02:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:02:52.282 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:02:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:02:53.674 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:02:53.702 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:02:53.735 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:02:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:02:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:54.189 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:02:54.219 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:02:54.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:02:54.760 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:02:54.792 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:54.864 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:02:54.894 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:02:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:57.827 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:02:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:02:57.962 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:02:57.962 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:02:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:02:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:03:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:03:10.083 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-04-23T18:03:10.083 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-04-23T18:03:10.083 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-04-23T18:03:10.084 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-04-23T18:03:10.084 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-04-23T18:03:10.084 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-04-23T18:03:10.084 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-04-23T18:03:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:10.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:03:10.288 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:03:10.288 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:10.288 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:03:10.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:11.816 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:03:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:03:11.843 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:11.843 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:03:11.843 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:03:11.844 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:03:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:03:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:03:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:03:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:03:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:03:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:03:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:03:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:03:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:03:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:03:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:03:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:03:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:03:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:03:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:03:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:03:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:03:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:03:15.114 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:03:15.151 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:03:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:03:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:03:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:03:16.480 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:03:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:03:16.535 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:03:16.535 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:16.536 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:03:16.536 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:03:16.536 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:03:16.536 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:16.587 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:03:16.598 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:03:16.700 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:03:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:03:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:03:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:03:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:17.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:03:17.863 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:03:17.863 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:17.863 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:03:17.863 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:03:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:18.258 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:03:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:03:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:03:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:03:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:03:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:03:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:03:23.178 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:03:23.228 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:03:23.259 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:03:23.286 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:03:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:03:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:03:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:03:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:03:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:03:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:03:25.095 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:03:25.180 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:03:25.223 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:03:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:03:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:03:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:03:25.468 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:03:25.501 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:03:25.501 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:03:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:03:25.653 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:03:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:03:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:03:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:03:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:03:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:03:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:03:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:03:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:03:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:03:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:03:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:03:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:03:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:03:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:03:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:03:33.239 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.240 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.242 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:03:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:03:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:03:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:03:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:03:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:03:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:03:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:03:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:03:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:03:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:03:33.254 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-23T18:03:33.649 DEBUG:teuthology.parallel:result is None 2024-04-23T18:03:33.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:33.794 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:03:33.821 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:33.822 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:33.823 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:03:33.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:34.210 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:03:34.237 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:34.238 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:34.238 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:03:34.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:34.381 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:03:34.403 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:34.403 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:03:34.403 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:03:34.405 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T18:03:34.405 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:03:34.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:03:34.438 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:03:34.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:03:34.471 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:03:34.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:03:34.500 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T18:03:34.501 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:03:34.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:03:34.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:03:34.600 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:03:34.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:03:34.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:03:34.698 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:03:34.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:03:34.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:03:34.790 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T18:03:34.790 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:03:34.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:03:34.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:03:34.886 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:03:34.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:03:34.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:03:34.984 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:03:34.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:03:35.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:03:35.074 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T18:03:35.075 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:03:35.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:03:35.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:03:35.165 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:03:35.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:03:35.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:03:35.264 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:03:35.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:03:35.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:03:35.354 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T18:03:35.465 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:03:35.465 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:35.466 INFO:tasks.cephadm:Cluster fsid is cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:03:35.466 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T18:03:35.466 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.32', 'mon.b': '172.21.15.40', 'mon.c': '172.21.15.64'} 2024-04-23T18:03:35.466 INFO:tasks.cephadm:First mon is mon.a on smithi032 2024-04-23T18:03:35.466 INFO:tasks.cephadm:First mgr is x 2024-04-23T18:03:35.466 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T18:03:35.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-23T18:03:35.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-04-23T18:03:35.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-04-23T18:03:35.546 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T18:03:35.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:35.693 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T18:03:36.015 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:36.017 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-23T18:03:36.017 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:37.136 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:03 /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:37.137 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:37.967 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:03 /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:37.968 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:38.933 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:03 /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:38.933 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:38.953 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:38.973 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:03:39.009 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-23T18:03:39.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:03:39.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:03:39.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:03:40.480 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:03:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:03:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:04:12.061 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-23T18:04:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:04:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:04:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-04-23T18:04:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:04:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-23T18:04:12.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-23T18:04:41.584 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:04:41.584 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:04:41.584 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:04:41.585 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-23T18:04:41.585 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:04:41.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:04:41.585 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:04:41.813 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-23T18:04:41.814 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-23T18:04:41.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-23T18:04:41.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-04-23T18:04:41.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-04-23T18:04:41.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-23T18:04:41.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-04-23T18:04:41.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-04-23T18:04:42.010 INFO:tasks.cephadm:Writing seed config... 2024-04-23T18:04:42.011 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-23T18:04:42.012 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-23T18:04:42.013 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T18:04:42.014 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T18:04:42.017 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-23T18:04:42.017 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T18:04:42.018 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-23T18:04:42.019 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:04:42.019 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T18:04:42.035 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cdb712f8-019b-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-23T18:04:42.035 DEBUG:teuthology.orchestra.run.smithi032:mon.a> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T18:04:42.078 DEBUG:teuthology.orchestra.run.smithi032:mgr.x> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T18:04:42.120 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T18:04:42.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:04:42.317 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:04:42.317 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', 'cdb712f8-019b-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-04-23T18:04:42.317 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T18:04:42.317 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-04-23T18:04:42.362 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:04:42.362 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-04-23T18:04:42.363 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-04-23T18:04:42.370 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:04:42.370 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:04:42.376 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:04:42.376 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:04:42.382 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-23T18:04:42.389 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-23T18:04:42.389 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:04:42.389 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-04-23T18:04:42.413 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:04:42.413 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T18:04:42.413 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-04-23T18:04:42.413 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-04-23T18:04:42.419 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:04:42.419 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:04:42.426 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:04:42.426 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:04:42.431 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 139741413518496 on /run/cephadm/cdb712f8-019b-11ef-bc93-c7b262605968.lock 2024-04-23T18:04:42.438 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 139741413518496 acquired on /run/cephadm/cdb712f8-019b-11ef-bc93-c7b262605968.lock 2024-04-23T18:04:42.439 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2024-04-23T18:04:42.439 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-04-23T18:04:42.439 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-23T18:04:42.442 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-04-23T18:04:42.442 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-04-23T18:04:42.445 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T18:04:42.445 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-23T18:04:42.445 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-23T18:04:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T18:04:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T18:04:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:04:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-23T18:04:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-04-23T18:04:42.448 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:04:42.448 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-23T18:04:42.448 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-23T18:04:42.448 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-23T18:04:42.449 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T18:04:42.450 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:04:43.178 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:04:43.178 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:04:43.178 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T18:04:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-23T18:04:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T18:04:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:04:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T18:04:44.255 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:04:44.256 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:04:44.256 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T18:04:45.227 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-04-23T18:04:45.228 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-04-23T18:04:46.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQA9+CdmNwVlIhAAZ2l0hOgtlEleXN40BPQMaA== 2024-04-23T18:04:47.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQA++CdmNwPvIhAAMid7VgKUYuRzw7Dz4Vvnrw== 2024-04-23T18:04:48.266 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQA/+CdmnhZJIxAAXL9LrTycUVE0RIx7k6uv6g== 2024-04-23T18:04:48.267 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for a [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:04:49.189 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-04-23T18:04:50.102 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.a on 2024-04-23T18:04:50.323 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-23T18:04:50.473 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T18:04:50.670 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target. 2024-04-23T18:04:50.670 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target. 2024-04-23T18:04:50.914 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:04:50.915 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T18:04:51.090 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target.wants/ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968@.service. 2024-04-23T18:04:51.696 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:51 smithi032 ceph-mon[28844]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:04:51.696 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:51 smithi032 bash[28825]: 232d508a747541ee461df86a0ce3b1fbbd5da0a51e601c7cbf46590f79f05920 2024-04-23T18:04:51.709 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:04:51.709 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-23T18:04:51.716 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:04:51.716 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:04:51.716 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-23T18:04:51.716 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:04:51.717 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-04-23T18:04:51.717 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-04-23T18:04:51.959 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:51 smithi032 systemd[1]: Started Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: monmap epoch 1 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: last_changed 2024-04-23T18:04:48.613148+0000 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: min_mon_release 19 (squid) 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: election_strategy: 1 2024-04-23T18:04:52.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:04:52.796 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: fsmap 2024-04-23T18:04:52.796 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:04:52.796 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: mgrmap e1: no daemons active 2024-04-23T18:04:52.796 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:52 smithi032 ceph-mon[28844]: from='client.? 172.21.15.32:0/2868014134' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T18:04:52.797 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.692476s) 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:52.804 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.163+0000 7fc63ce62700 1 Processor -- start 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.163+0000 7fc63ce62700 1 -- start start 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63ce62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63ce62700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc638108da0 con 0x7fc6381083b0 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56140/0 (socket says 172.21.15.32:56140) 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63659c700 1 -- 172.21.15.32:0/1460907747 learned_addr learned my addr 172.21.15.32:0/1460907747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:52.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.164+0000 7fc63659c700 1 -- 172.21.15.32:0/1460907747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc638109560 con 0x7fc6381083b0 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63659c700 1 --2- 172.21.15.32:0/1460907747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc630009e80 tx=0x7fc63000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38f421c16412ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63559a700 1 -- 172.21.15.32:0/1460907747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc630017070 con 0x7fc6381083b0 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63559a700 1 -- 172.21.15.32:0/1460907747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc630003a20 con 0x7fc6381083b0 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63559a700 1 -- 172.21.15.32:0/1460907747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc630003d20 con 0x7fc6381083b0 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 -- 172.21.15.32:0/1460907747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 msgr2=0x7fc6381087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 --2- 172.21.15.32:0/1460907747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc630009e80 tx=0x7fc63000b330 comp rx=0 tx=0).stop 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 -- 172.21.15.32:0/1460907747 shutdown_connections 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 --2- 172.21.15.32:0/1460907747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc6381087d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 -- 172.21.15.32:0/1460907747 >> 172.21.15.32:0/1460907747 conn(0x7fc6381039b0 msgr2=0x7fc638105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 -- 172.21.15.32:0/1460907747 shutdown_connections 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.165+0000 7fc63ce62700 1 -- 172.21.15.32:0/1460907747 wait complete. 2024-04-23T18:04:52.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.166+0000 7fc63ce62700 1 Processor -- start 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.166+0000 7fc63ce62700 1 -- start start 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.166+0000 7fc63ce62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63ce62700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc638109a90 con 0x7fc6381083b0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56152/0 (socket says 172.21.15.32:56152) 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63659c700 1 -- 172.21.15.32:0/2868014134 learned_addr learned my addr 172.21.15.32:0/2868014134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63659c700 1 -- 172.21.15.32:0/2868014134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63819bf40 con 0x7fc6381083b0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63659c700 1 --2- 172.21.15.32:0/2868014134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc630004000 tx=0x7fc630004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc630022020 con 0x7fc6381083b0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc630018040 con 0x7fc6381083b0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63819c1d0 con 0x7fc6381083b0 2024-04-23T18:04:52.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc630017070 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63819eec0 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc630025020 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.167+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc63000f050 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.168+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6380640a0 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.171+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc63002c030 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.243+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fc63802e140 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.243+0000 7fc62f7fe700 1 -- 172.21.15.32:0/2868014134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fc63001e480 con 0x7fc6381083b0 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.244+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 msgr2=0x7fc63819ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.244+0000 7fc63ce62700 1 --2- 172.21.15.32:0/2868014134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc630004000 tx=0x7fc630004ad0 comp rx=0 tx=0).stop 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.244+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 shutdown_connections 2024-04-23T18:04:52.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.244+0000 7fc63ce62700 1 --2- 172.21.15.32:0/2868014134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6381083b0 0x7fc63819ba00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.244+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 >> 172.21.15.32:0/2868014134 conn(0x7fc6381039b0 msgr2=0x7fc6381926c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.245+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 shutdown_connections 2024-04-23T18:04:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:52.245+0000 7fc63ce62700 1 -- 172.21.15.32:0/2868014134 wait complete. 2024-04-23T18:04:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-04-23T18:04:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T18:04:53.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:53 smithi032 ceph-mon[28844]: from='client.? 172.21.15.32:0/3487501286' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:04:53.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:53 smithi032 ceph-mon[28844]: from='client.? 172.21.15.32:0/3487501286' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T18:04:53.863 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:53.863 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:04:53.863 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:04:53.863 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:04:53.863 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:04:53.864 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b2dce700 1 Processor -- start 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b2dce700 1 -- start start 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b2dce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b2dce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac07be20 con 0x7f85ac106c60 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b0b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b0b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56166/0 (socket says 172.21.15.32:56166) 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.230+0000 7f85b0b6a700 1 -- 172.21.15.32:0/707147554 learned_addr learned my addr 172.21.15.32:0/707147554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85b0b6a700 1 -- 172.21.15.32:0/707147554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac1075c0 con 0x7f85ac106c60 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85b0b6a700 1 --2- 172.21.15.32:0/707147554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f85a00098a0 tx=0x7f85a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a356bfb59b2fffb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85ab7fe700 1 -- 172.21.15.32:0/707147554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85a0004160 con 0x7f85ac106c60 2024-04-23T18:04:53.865 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85ab7fe700 1 -- 172.21.15.32:0/707147554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0004300 con 0x7f85ac106c60 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85ab7fe700 1 -- 172.21.15.32:0/707147554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85a00036a0 con 0x7f85ac106c60 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85b2dce700 1 -- 172.21.15.32:0/707147554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 msgr2=0x7f85ac107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.231+0000 7f85b2dce700 1 --2- 172.21.15.32:0/707147554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f85a00098a0 tx=0x7f85a0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- 172.21.15.32:0/707147554 shutdown_connections 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 --2- 172.21.15.32:0/707147554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac107080 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- 172.21.15.32:0/707147554 >> 172.21.15.32:0/707147554 conn(0x7f85ac102860 msgr2=0x7f85ac104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- 172.21.15.32:0/707147554 shutdown_connections 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- 172.21.15.32:0/707147554 wait complete. 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 Processor -- start 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- start start 2024-04-23T18:04:53.866 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.232+0000 7f85b2dce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac107af0 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b0b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b0b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56176/0 (socket says 172.21.15.32:56176) 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b0b6a700 1 -- 172.21.15.32:0/3487501286 learned_addr learned my addr 172.21.15.32:0/3487501286 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b0b6a700 1 -- 172.21.15.32:0/3487501286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac19e080 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b0b6a700 1 --2- 172.21.15.32:0/3487501286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85a0000c00 tx=0x7f85a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85a0003db0 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0011630 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac19e310 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85a0011950 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.233+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1a1000 con 0x7f85ac106c60 2024-04-23T18:04:53.867 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.234+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f85a0011af0 con 0x7f85ac106c60 2024-04-23T18:04:53.868 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.234+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f85a001c450 con 0x7f85ac106c60 2024-04-23T18:04:53.868 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.234+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac0640a0 con 0x7f85ac106c60 2024-04-23T18:04:53.868 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.237+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f85a0032020 con 0x7f85ac106c60 2024-04-23T18:04:53.868 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.307+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f85ac1a11d0 con 0x7f85ac106c60 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.310+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f85a0004260 con 0x7f85ac106c60 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.310+0000 7f85a9ffb700 1 -- 172.21.15.32:0/3487501286 <== mon.0 v2:172.21.15.32:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f85a001c6a0 con 0x7f85ac106c60 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 msgr2=0x7f85ac19db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 --2- 172.21.15.32:0/3487501286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85a0000c00 tx=0x7f85a0004750 comp rx=0 tx=0).stop 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 shutdown_connections 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 --2- 172.21.15.32:0/3487501286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ac106c60 0x7f85ac19db40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 >> 172.21.15.32:0/3487501286 conn(0x7f85ac102860 msgr2=0x7f85ac194180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 shutdown_connections 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:53.311+0000 7f85b2dce700 1 -- 172.21.15.32:0/3487501286 wait complete. 2024-04-23T18:04:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-04-23T18:04:54.803 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:54 smithi032 ceph-mon[28844]: from='client.? 172.21.15.32:0/3237202219' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f697db2a700 1 Processor -- start 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f697db2a700 1 -- start start 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f697db2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f697db2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6978108da0 con 0x7f69781083b0 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f69777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f69777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56184/0 (socket says 172.21.15.32:56184) 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.270+0000 7f69777fe700 1 -- 172.21.15.32:0/1923297706 learned_addr learned my addr 172.21.15.32:0/1923297706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f69777fe700 1 -- 172.21.15.32:0/1923297706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6978109560 con 0x7f69781083b0 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f69777fe700 1 --2- 172.21.15.32:0/1923297706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f69680098a0 tx=0x7f6968009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71253c95c2c601aa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f69767fc700 1 -- 172.21.15.32:0/1923297706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6968004160 con 0x7f69781083b0 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f69767fc700 1 -- 172.21.15.32:0/1923297706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6968004300 con 0x7f69781083b0 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f69767fc700 1 -- 172.21.15.32:0/1923297706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6968011af0 con 0x7f69781083b0 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f697db2a700 1 -- 172.21.15.32:0/1923297706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 msgr2=0x7f69781087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f697db2a700 1 --2- 172.21.15.32:0/1923297706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f69680098a0 tx=0x7f6968009bb0 comp rx=0 tx=0).stop 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.271+0000 7f697db2a700 1 -- 172.21.15.32:0/1923297706 shutdown_connections 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 --2- 172.21.15.32:0/1923297706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f69781087d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 -- 172.21.15.32:0/1923297706 >> 172.21.15.32:0/1923297706 conn(0x7f69781039b0 msgr2=0x7f6978105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 -- 172.21.15.32:0/1923297706 shutdown_connections 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 -- 172.21.15.32:0/1923297706 wait complete. 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 Processor -- start 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.272+0000 7f697db2a700 1 -- start start 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f697db2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f697db2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f697810c910 con 0x7f69781083b0 2024-04-23T18:04:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f69777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f69777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56188/0 (socket says 172.21.15.32:56188) 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f69777fe700 1 -- 172.21.15.32:0/3237202219 learned_addr learned my addr 172.21.15.32:0/3237202219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f69777fe700 1 -- 172.21.15.32:0/3237202219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697819c790 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f69777fe700 1 --2- 172.21.15.32:0/3237202219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6968003f40 tx=0x7f6968004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69680036a0 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6968024430 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f696801a5c0 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697819ca20 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.273+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697819f710 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.274+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f696801ac50 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.274+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6968019700 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.274+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69780640a0 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.277+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f69680245e0 con 0x7f69781083b0 2024-04-23T18:04:55.072 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.348+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f697819fc50 con 0x7f69781083b0 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.348+0000 7f6974ff9700 1 -- 172.21.15.32:0/3237202219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f696802dcb0 con 0x7f69781083b0 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 msgr2=0x7f697819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 --2- 172.21.15.32:0/3237202219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6968003f40 tx=0x7f6968004790 comp rx=0 tx=0).stop 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 shutdown_connections 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 --2- 172.21.15.32:0/3237202219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69781083b0 0x7f697819c250 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 >> 172.21.15.32:0/3237202219 conn(0x7f69781039b0 msgr2=0x7f69781044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 shutdown_connections 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:04:54.349+0000 7f697db2a700 1 -- 172.21.15.32:0/3237202219 wait complete. 2024-04-23T18:04:55.073 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-04-23T18:04:55.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 systemd[1]: Stopping Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:04:55.580 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a[28840]: 2024-04-23T18:04:55.329+0000 7f8afe585700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:04:55.580 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a[28840]: 2024-04-23T18:04:55.329+0000 7f8afe585700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T18:04:55.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 podman[29090]: 2024-04-23 18:04:55.586353933 +0000 UTC m=+0.333874320 container died 232d508a747541ee461df86a0ce3b1fbbd5da0a51e601c7cbf46590f79f05920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-23T18:04:55.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 podman[29090]: 2024-04-23 18:04:55.881109648 +0000 UTC m=+0.628630041 container cleanup 232d508a747541ee461df86a0ce3b1fbbd5da0a51e601c7cbf46590f79f05920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:04:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:55 smithi032 bash[29090]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:04:57.570 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:57 smithi032 podman[29104]: 2024-04-23 18:04:57.307894938 +0000 UTC m=+1.715047816 container remove 232d508a747541ee461df86a0ce3b1fbbd5da0a51e601c7cbf46590f79f05920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:04:57.570 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:57 smithi032 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T18:04:57.570 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:57 smithi032 systemd[1]: Stopped Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:04:57.570 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:57 smithi032 systemd[1]: Starting Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:04:58.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:57 smithi032 podman[29202]: 2024-04-23 18:04:57.644436791 +0000 UTC m=+0.018611083 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:04:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:04:58 smithi032 podman[29202]: 2024-04-23 18:04:58.779111124 +0000 UTC m=+1.153285401 container create 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 podman[29202]: 2024-04-23 18:05:00.371758198 +0000 UTC m=+2.745932476 container init 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 podman[29202]: 2024-04-23 18:05:00.376172985 +0000 UTC m=+2.750347262 container start 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: pidfile_write: ignore empty --pid-file 2024-04-23T18:05:00.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: load: jerasure load: lrc 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Git sha 0 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: DB SUMMARY 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: DB Session ID: 4PX76KIEPQ29RJXO4428 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104534 ; 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:05:00.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.env: 0x55fc3b836cc0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.info_log: 0x55fc3c157be0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.statistics: (nil) 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.use_fsync: 0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:05:00.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.db_log_dir: 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.wal_dir: 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:05:00.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.write_buffer_manager: 0x55fc3c0eb0e0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.unordered_write: 0 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:05:00.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:05:00.764 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.row_cache: None 2024-04-23T18:05:00.764 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.wal_filter: None 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.wal_compression: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:05:00.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:05:00.766 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_open_files: -1 2024-04-23T18:05:00.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Compression algorithms supported: 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kZSTD supported: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:05:00.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.merge_operator: 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_filter: None 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fc3c157dc0) 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:05:00.769 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_type: 0 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_index_type: 0 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_shortening: 1 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: checksum: 4 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: no_block_cache: 0 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache: 0x55fc3c11d1f0 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_options: 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: capacity : 536870912 2024-04-23T18:05:00.770 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_shard_bits : 4 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: strict_capacity_limit : 0 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_compressed: (nil) 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: persistent_cache: (nil) 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size: 4096 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size_deviation: 10 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_restart_interval: 16 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_block_restart_interval: 1 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: metadata_block_size: 4096 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: partition_filters: 0 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: use_delta_encoding: 1 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: filter_policy: bloomfilter 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: whole_key_filtering: 1 2024-04-23T18:05:00.771 INFO:journalctl@ceph.mon.a.smithi032.stdout: verify_compression: 0 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: format_version: 5 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_index_compression: 1 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_align: 0 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: max_auto_readahead_size: 262144 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: prepopulate_block_cache: 0 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression: NoCompression 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:05:00.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.num_levels: 7 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:05:00.773 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:05:00.774 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:05:00.775 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:05:00.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:05:00.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:05:00.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:05:00.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:05:00.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.ttl: 2592000 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:05:00.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a4d1367e-9283-4720-8d8f-3fd3ee4d78d9 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895500408336, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895500409545, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99761, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97643, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 90001, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895500, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a4d1367e-9283-4720-8d8f-3fd3ee4d78d9", "db_session_id": "4PX76KIEPQ29RJXO4428", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895500409640, "job": 1, "event": "recovery_finished"} 2024-04-23T18:05:00.779 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fc3c232000 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: DB pointer 0x55fc3c21c000 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** DB Stats ** 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.780 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: L0 2/0 99.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Sum 2/0 99.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.781 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative compaction: 0.00 GB write, 26.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval compaction: 0.00 GB write, 26.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache BinnedLRUCache@0x55fc3c11d1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:05:00.782 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: starting mon.a rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???) e1 preinit fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).mds e1 new map 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).mds e1 print_map 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: e1 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: legacy client fscid: -1 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout: No filesystems configured 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:05:00.783 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: monmap epoch 1 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: last_changed 2024-04-23T18:04:48.613148+0000 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: min_mon_release 19 (squid) 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: election_strategy: 1 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: fsmap 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:05:00.784 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:00 smithi032 ceph-mon[29221]: mgrmap e1: no daemons active 2024-04-23T18:05:01.147 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T18:05:01.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:01 smithi032 bash[29202]: 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f 2024-04-23T18:05:01.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:01 smithi032 systemd[1]: Started Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:05:05.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2617878286' entity='client.admin' 2024-04-23T18:05:07.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.057+0000 7fe0abaa5700 1 Processor -- start 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0abaa5700 1 -- start start 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0abaa5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0abaa5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4108da0 con 0x7fe0a41083b0 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0a9841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0a9841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37142/0 (socket says 172.21.15.32:37142) 2024-04-23T18:05:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.058+0000 7fe0a9841700 1 -- 172.21.15.32:0/1111527672 learned_addr learned my addr 172.21.15.32:0/1111527672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.059+0000 7fe0a9841700 1 -- 172.21.15.32:0/1111527672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a4109560 con 0x7fe0a41083b0 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.059+0000 7fe0a9841700 1 --2- 172.21.15.32:0/1111527672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e80 tx=0x7fe09400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0036aac80bbc0b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.059+0000 7fe0a883f700 1 -- 172.21.15.32:0/1111527672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe094017070 con 0x7fe0a41083b0 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.059+0000 7fe0a883f700 1 -- 172.21.15.32:0/1111527672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe09400b8c0 con 0x7fe0a41083b0 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.059+0000 7fe0a883f700 1 -- 172.21.15.32:0/1111527672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe094004490 con 0x7fe0a41083b0 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 -- 172.21.15.32:0/1111527672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 msgr2=0x7fe0a41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 --2- 172.21.15.32:0/1111527672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e80 tx=0x7fe09400b330 comp rx=0 tx=0).stop 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 -- 172.21.15.32:0/1111527672 shutdown_connections 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 --2- 172.21.15.32:0/1111527672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a41087d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 -- 172.21.15.32:0/1111527672 >> 172.21.15.32:0/1111527672 conn(0x7fe0a41039b0 msgr2=0x7fe0a4105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 -- 172.21.15.32:0/1111527672 shutdown_connections 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 -- 172.21.15.32:0/1111527672 wait complete. 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.060+0000 7fe0abaa5700 1 Processor -- start 2024-04-23T18:05:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0abaa5700 1 -- start start 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0abaa5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0abaa5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a410c910 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a9841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a9841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37154/0 (socket says 172.21.15.32:37154) 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a9841700 1 -- 172.21.15.32:0/2617878286 learned_addr learned my addr 172.21.15.32:0/2617878286 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a9841700 1 -- 172.21.15.32:0/2617878286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a419c6f0 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a9841700 1 --2- 172.21.15.32:0/2617878286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe09400b7b0 tx=0x7fe09400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe094020020 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.061+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe094012430 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a419c980 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe094017040 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a419f670 con 0x7fe0a41083b0 2024-04-23T18:05:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe094023020 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe09400f050 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.062+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a40640a0 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.065+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe09402a020 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.136+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe0a419f840 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0a2ffd700 1 -- 172.21.15.32:0/2617878286 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe0940125e0 con 0x7fe0a41083b0 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 msgr2=0x7fe0a419c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0abaa5700 1 --2- 172.21.15.32:0/2617878286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe09400b7b0 tx=0x7fe09400be20 comp rx=0 tx=0).stop 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 shutdown_connections 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0abaa5700 1 --2- 172.21.15.32:0/2617878286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0a41083b0 0x7fe0a419c1b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.138+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 >> 172.21.15.32:0/2617878286 conn(0x7fe0a41039b0 msgr2=0x7fe0a4104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.139+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 shutdown_connections 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:04.139+0000 7fe0abaa5700 1 -- 172.21.15.32:0/2617878286 wait complete. 2024-04-23T18:05:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T18:05:07.520 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:05:07.520 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-04-23T18:05:07.520 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T18:05:07.521 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T18:05:07.752 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:05:07.753 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-23T18:05:07.901 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968.target.wants/ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-cdb712f8-019b-11ef-bc93-c7b262605968@.service. 2024-04-23T18:05:08.441 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:05:08.442 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-23T18:05:08.449 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:05:08.449 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:05:08.449 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-23T18:05:08.449 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:05:08.458 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:05:08.458 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-23T18:05:08.465 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:05:08.465 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:05:08.466 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-23T18:05:08.466 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T18:05:08.466 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-04-23T18:05:08.466 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-04-23T18:05:09.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:09 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/881948480' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:05:09.259 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:09.176+0000 7fabbe3b2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:05:09.556 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:09.268+0000 7fabbe3b2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:05:09.570 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:05:09.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:05:09.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:05:09.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:04:51.553554+0000", 2024-04-23T18:05:09.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.911+0000 7f50c5f56700 1 Processor -- start 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.911+0000 7f50c5f56700 1 -- start start 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50c5f56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50c5f56700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f50c0107c40 con 0x7f50c0107250 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40676/0 (socket says 172.21.15.32:40676) 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50bf7fe700 1 -- 172.21.15.32:0/533090934 learned_addr learned my addr 172.21.15.32:0/533090934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50bf7fe700 1 -- 172.21.15.32:0/533090934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c0108450 con 0x7f50c0107250 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.912+0000 7f50bf7fe700 1 --2- 172.21.15.32:0/533090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f50b00098a0 tx=0x7f50b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b8a662da86d3a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50be7fc700 1 -- 172.21.15.32:0/533090934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50b0004160 con 0x7f50c0107250 2024-04-23T18:05:09.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50be7fc700 1 -- 172.21.15.32:0/533090934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50b0004300 con 0x7f50c0107250 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50be7fc700 1 -- 172.21.15.32:0/533090934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50b0011af0 con 0x7f50c0107250 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 -- 172.21.15.32:0/533090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 msgr2=0x7f50c0107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 --2- 172.21.15.32:0/533090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f50b00098a0 tx=0x7f50b0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 -- 172.21.15.32:0/533090934 shutdown_connections 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 --2- 172.21.15.32:0/533090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c0107670 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 -- 172.21.15.32:0/533090934 >> 172.21.15.32:0/533090934 conn(0x7f50c0102890 msgr2=0x7f50c0104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 -- 172.21.15.32:0/533090934 shutdown_connections 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.913+0000 7f50c5f56700 1 -- 172.21.15.32:0/533090934 wait complete. 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.914+0000 7f50c5f56700 1 Processor -- start 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.914+0000 7f50c5f56700 1 -- start start 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.914+0000 7f50c5f56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:09.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.914+0000 7f50c5f56700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f50c010b800 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40686/0 (socket says 172.21.15.32:40686) 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bf7fe700 1 -- 172.21.15.32:0/881948480 learned_addr learned my addr 172.21.15.32:0/881948480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bf7fe700 1 -- 172.21.15.32:0/881948480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c019ef70 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bf7fe700 1 --2- 172.21.15.32:0/881948480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f50b0003f40 tx=0x7f50b0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50b00038b0 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50b0003a50 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c019f140 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50b001a5c0 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.915+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c019f4e0 con 0x7f50c0107250 2024-04-23T18:05:09.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.916+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f50b001ac50 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.916+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f50b0019a40 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.917+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c00640a0 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.921+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f50b0019ce0 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.995+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f50c019f7e0 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.996+0000 7f50bcff9700 1 -- 172.21.15.32:0/881948480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f50b0024430 con 0x7f50c0107250 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 msgr2=0x7f50c007ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 --2- 172.21.15.32:0/881948480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f50b0003f40 tx=0x7f50b0011890 comp rx=0 tx=0).stop 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 shutdown_connections 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 --2- 172.21.15.32:0/881948480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50c0107250 0x7f50c007ba40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:09.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 >> 172.21.15.32:0/881948480 conn(0x7f50c0102890 msgr2=0x7f50c01032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:09.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 shutdown_connections 2024-04-23T18:05:09.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:08.997+0000 7f50c5f56700 1 -- 172.21.15.32:0/881948480 wait complete. 2024-04-23T18:05:09.580 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-04-23T18:05:10.881 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:10.534+0000 7fabbe3b2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:05:10.882 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:10.710+0000 7fabbe3b2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:10.882 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:10.886+0000 7fabbe3b2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:05:11.466 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:11.187+0000 7fabbe3b2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:05:11.466 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:11.284+0000 7fabbe3b2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:05:11.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:11.499+0000 7fabbe3b2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:05:11.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:11.705+0000 7fabbe3b2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:05:12.622 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:12.359+0000 7fabbe3b2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:05:12.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1751491109' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:05:12.911 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:12.802+0000 7fabbe3b2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:05:13.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:12.915+0000 7fabbe3b2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:05:13.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:13.127+0000 7fabbe3b2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:05:13.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:13.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:13.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:05:13.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:05:13.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:05:13.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:05:13.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:05:13.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:04:51.553554+0000", 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:05:13.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa821537700 1 Processor -- start 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa821537700 1 -- start start 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa821537700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa821537700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c107c70 con 0x7fa81c107280 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa81affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa81affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40694/0 (socket says 172.21.15.32:40694) 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa81affd700 1 -- 172.21.15.32:0/691161232 learned_addr learned my addr 172.21.15.32:0/691161232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.730+0000 7fa81affd700 1 -- 172.21.15.32:0/691161232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c108480 con 0x7fa81c107280 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa81affd700 1 --2- 172.21.15.32:0/691161232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0098a0 tx=0x7fa80c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f15277b4f540aec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa819ffb700 1 -- 172.21.15.32:0/691161232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c004160 con 0x7fa81c107280 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa819ffb700 1 -- 172.21.15.32:0/691161232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa80c004300 con 0x7fa81c107280 2024-04-23T18:05:13.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa819ffb700 1 -- 172.21.15.32:0/691161232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c011af0 con 0x7fa81c107280 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa821537700 1 -- 172.21.15.32:0/691161232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 msgr2=0x7fa81c1076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa821537700 1 --2- 172.21.15.32:0/691161232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0098a0 tx=0x7fa80c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa821537700 1 -- 172.21.15.32:0/691161232 shutdown_connections 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa821537700 1 --2- 172.21.15.32:0/691161232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c1076a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.731+0000 7fa821537700 1 -- 172.21.15.32:0/691161232 >> 172.21.15.32:0/691161232 conn(0x7fa81c102860 msgr2=0x7fa81c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 -- 172.21.15.32:0/691161232 shutdown_connections 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 -- 172.21.15.32:0/691161232 wait complete. 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 Processor -- start 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 -- start start 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.732+0000 7fa821537700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c10b830 con 0x7fa81c107280 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa81affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:13.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa81affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40706/0 (socket says 172.21.15.32:40706) 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa81affd700 1 -- 172.21.15.32:0/1751491109 learned_addr learned my addr 172.21.15.32:0/1751491109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa81affd700 1 -- 172.21.15.32:0/1751491109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c19c780 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa81affd700 1 --2- 172.21.15.32:0/1751491109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa80c000c00 tx=0x7fa80c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c003b20 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c19ca10 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa80c01a8c0 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c024940 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.733+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c19f700 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.734+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa80c022020 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.734+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa80c0193c0 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.734+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c0640a0 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.737+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa80c02dde0 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.811+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa81c19fc40 con 0x7fa81c107280 2024-04-23T18:05:13.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.811+0000 7fa803fff700 1 -- 172.21.15.32:0/1751491109 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fa80c01aa70 con 0x7fa81c107280 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 msgr2=0x7fa81c19c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 --2- 172.21.15.32:0/1751491109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa80c000c00 tx=0x7fa80c011b70 comp rx=0 tx=0).stop 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 shutdown_connections 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 --2- 172.21.15.32:0/1751491109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c107280 0x7fa81c19c240 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 >> 172.21.15.32:0/1751491109 conn(0x7fa81c102860 msgr2=0x7fa81c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 shutdown_connections 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:12.813+0000 7fa821537700 1 -- 172.21.15.32:0/1751491109 wait complete. 2024-04-23T18:05:13.381 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2024-04-23T18:05:14.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:13.688+0000 7fabbe3b2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:05:14.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:13.783+0000 7fabbe3b2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:05:14.390 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:14.037+0000 7fabbe3b2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:05:14.391 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:14.175+0000 7fabbe3b2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:05:14.391 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:14.274+0000 7fabbe3b2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:05:14.391 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:14.395+0000 7fabbe3b2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:15.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:14.725+0000 7fabbe3b2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:05:15.605 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:15.354+0000 7fabbe3b2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:05:15.605 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:15.579+0000 7fabbe3b2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: Activating manager daemon x 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: mgrmap e2: x(active, starting, since 0.00323132s) 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:05:15.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: Manager daemon x is now available 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' 2024-04-23T18:05:15.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:15 smithi032 ceph-mon[29221]: from='mgr.14100 172.21.15.32:0/1365789455' entity='mgr.x' 2024-04-23T18:05:16.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:05:16.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:05:16.463 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:05:16.464 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:05:16.465 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:04:51.553554+0000", 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:16.466 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7af5480700 1 Processor -- start 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7af5480700 1 -- start start 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7af5480700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7af5480700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0108d90 con 0x7f7af01083a0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aeeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aeeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40778/0 (socket says 172.21.15.32:40778) 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aeeffd700 1 -- 172.21.15.32:0/1818603576 learned_addr learned my addr 172.21.15.32:0/1818603576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aeeffd700 1 -- 172.21.15.32:0/1818603576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0109550 con 0x7f7af01083a0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aeeffd700 1 --2- 172.21.15.32:0/1818603576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00098a0 tx=0x7f7ae0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff63a703d1492b42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.801+0000 7f7aee7fc700 1 -- 172.21.15.32:0/1818603576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0004160 con 0x7f7af01083a0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7aee7fc700 1 -- 172.21.15.32:0/1818603576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7ae0004300 con 0x7f7af01083a0 2024-04-23T18:05:16.467 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7aee7fc700 1 -- 172.21.15.32:0/1818603576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0011af0 con 0x7f7af01083a0 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 -- 172.21.15.32:0/1818603576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 msgr2=0x7f7af01087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 --2- 172.21.15.32:0/1818603576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00098a0 tx=0x7f7ae0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 -- 172.21.15.32:0/1818603576 shutdown_connections 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 --2- 172.21.15.32:0/1818603576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af01087c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 -- 172.21.15.32:0/1818603576 >> 172.21.15.32:0/1818603576 conn(0x7f7af0103940 msgr2=0x7f7af0105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 -- 172.21.15.32:0/1818603576 shutdown_connections 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.802+0000 7f7af5480700 1 -- 172.21.15.32:0/1818603576 wait complete. 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7af5480700 1 Processor -- start 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7af5480700 1 -- start start 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7af5480700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7af5480700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010c900 con 0x7f7af01083a0 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7aeeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7aeeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40784/0 (socket says 172.21.15.32:40784) 2024-04-23T18:05:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7aeeffd700 1 -- 172.21.15.32:0/1405398691 learned_addr learned my addr 172.21.15.32:0/1405398691 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.803+0000 7f7aeeffd700 1 -- 172.21.15.32:0/1405398691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af019c750 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7aeeffd700 1 --2- 172.21.15.32:0/1405398691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0003b20 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7ae001a8c0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af019c9e0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0024940 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.804+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af019f6d0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.805+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7f7ae0022020 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.805+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af00640a0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.805+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7ae004d190 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.807+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7ae000b3e0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.881+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7af019f8a0 con 0x7f7af01083a0 2024-04-23T18:05:16.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.881+0000 7f7aecff9700 1 -- 172.21.15.32:0/1405398691 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f7af019f8a0 con 0x7f7af01083a0 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 msgr2=0x7f7af019c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 --2- 172.21.15.32:0/1405398691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae0011b70 comp rx=0 tx=0).stop 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 shutdown_connections 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 --2- 172.21.15.32:0/1405398691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af01083a0 0x7f7af019c210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 >> 172.21.15.32:0/1405398691 conn(0x7f7af0103940 msgr2=0x7f7af01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 shutdown_connections 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:15.882+0000 7f7af5480700 1 -- 172.21.15.32:0/1405398691 wait complete. 2024-04-23T18:05:16.470 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (3/15)... 2024-04-23T18:05:16.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1405398691' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:05:16.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:16 smithi032 ceph-mon[29221]: mgrmap e3: x(active, since 1.00765s) 2024-04-23T18:05:18.884 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:18 smithi032 ceph-mon[29221]: mgrmap e4: x(active, since 2s) 2024-04-23T18:05:19.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3624872677' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:05:19.917 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:19.917 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:19.917 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:05:19.917 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:05:19.917 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:19.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:05:19.919 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:05:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:04:51.553554+0000", 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f71a0a0a700 1 Processor -- start 2024-04-23T18:05:19.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f71a0a0a700 1 -- start start 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f71a0a0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f71a0a0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f719c07c8c0 con 0x7f719c07de40 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f719a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f719a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47692/0 (socket says 172.21.15.32:47692) 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.926+0000 7f719a59c700 1 -- 172.21.15.32:0/3521356311 learned_addr learned my addr 172.21.15.32:0/3521356311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f719a59c700 1 -- 172.21.15.32:0/3521356311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c07ca40 con 0x7f719c07de40 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f719a59c700 1 --2- 172.21.15.32:0/3521356311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f718c0098a0 tx=0x7f718c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2540b8f6e35c4443 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f719959a700 1 -- 172.21.15.32:0/3521356311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c004160 con 0x7f719c07de40 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f719959a700 1 -- 172.21.15.32:0/3521356311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f718c004300 con 0x7f719c07de40 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f719959a700 1 -- 172.21.15.32:0/3521356311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c011af0 con 0x7f719c07de40 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3521356311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 msgr2=0x7f719c07c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.927+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3521356311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f718c0098a0 tx=0x7f718c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:19.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3521356311 shutdown_connections 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3521356311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c07c2f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3521356311 >> 172.21.15.32:0/3521356311 conn(0x7f719c1039b0 msgr2=0x7f719c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3521356311 shutdown_connections 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3521356311 wait complete. 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 Processor -- start 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.928+0000 7f71a0a0a700 1 -- start start 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f71a0a0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f71a0a0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f719c10e900 con 0x7f719c07de40 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f719a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f719a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47704/0 (socket says 172.21.15.32:47704) 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f719a59c700 1 -- 172.21.15.32:0/3624872677 learned_addr learned my addr 172.21.15.32:0/3624872677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f719a59c700 1 -- 172.21.15.32:0/3624872677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c19e920 con 0x7f719c07de40 2024-04-23T18:05:19.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f719a59c700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f718c003f40 tx=0x7f718c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c0038b0 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c19ebb0 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f718c003a50 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.929+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c01a5c0 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.930+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c1a18a0 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.930+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f718c024430 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.931+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c0640a0 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.931+0000 7f718b7fe700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 0x7f718403ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.931+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f718c04da70 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.931+0000 7f7199d9b700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 0x7f718403ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.931+0000 7f7199d9b700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 0x7f718403ba90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7190006fd0 tx=0x7f7190006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:18.936+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f718c04d490 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.230+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f719c1a1a70 con 0x7f719c07de40 2024-04-23T18:05:19.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.230+0000 7f718b7fe700 1 -- 172.21.15.32:0/3624872677 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f719c1a1a70 con 0x7f719c07de40 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.233+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 msgr2=0x7f718403ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.233+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 0x7f718403ba90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7190006fd0 tx=0x7f7190006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.233+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 msgr2=0x7f719c19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.233+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f718c003f40 tx=0x7f718c011890 comp rx=0 tx=0).stop 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 shutdown_connections 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f71840395d0 0x7f718403ba90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 --2- 172.21.15.32:0/3624872677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f719c07de40 0x7f719c19e3e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 >> 172.21.15.32:0/3624872677 conn(0x7f719c1039b0 msgr2=0x7f719c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 shutdown_connections 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:19.234+0000 7f71a0a0a700 1 -- 172.21.15.32:0/3624872677 wait complete. 2024-04-23T18:05:19.925 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-04-23T18:05:21.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:20 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1359073224' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:05:21.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f9236999700 1 Processor -- start 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f9236999700 1 -- start start 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f9236999700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f9236999700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f923007c810 con 0x7f923007de20 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f922ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f922ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47708/0 (socket says 172.21.15.32:47708) 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.324+0000 7f922ffff700 1 -- 172.21.15.32:0/167284075 learned_addr learned my addr 172.21.15.32:0/167284075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:21.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.325+0000 7f922ffff700 1 -- 172.21.15.32:0/167284075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923007c990 con 0x7f923007de20 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.325+0000 7f922ffff700 1 --2- 172.21.15.32:0/167284075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f92200098a0 tx=0x7f9220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8e23f99180bbdb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.325+0000 7f922effd700 1 -- 172.21.15.32:0/167284075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9220004160 con 0x7f923007de20 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.325+0000 7f922effd700 1 -- 172.21.15.32:0/167284075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9220004300 con 0x7f923007de20 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.325+0000 7f922effd700 1 -- 172.21.15.32:0/167284075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9220011af0 con 0x7f923007de20 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 -- 172.21.15.32:0/167284075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 msgr2=0x7f923007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 --2- 172.21.15.32:0/167284075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f92200098a0 tx=0x7f9220009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 -- 172.21.15.32:0/167284075 shutdown_connections 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 --2- 172.21.15.32:0/167284075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923007c2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 -- 172.21.15.32:0/167284075 >> 172.21.15.32:0/167284075 conn(0x7f92301039b0 msgr2=0x7f9230105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 -- 172.21.15.32:0/167284075 shutdown_connections 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 -- 172.21.15.32:0/167284075 wait complete. 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.326+0000 7f9236999700 1 Processor -- start 2024-04-23T18:05:21.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f9236999700 1 -- start start 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f9236999700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f9236999700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f923010e920 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f922ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f922ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47712/0 (socket says 172.21.15.32:47712) 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f922ffff700 1 -- 172.21.15.32:0/1359073224 learned_addr learned my addr 172.21.15.32:0/1359073224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f922ffff700 1 -- 172.21.15.32:0/1359073224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923019e930 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.327+0000 7f922ffff700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9220003f40 tx=0x7f9220011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.328+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92200038b0 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.328+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9220003a50 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.328+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923019ebc0 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.328+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f922001a5c0 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.328+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92301a18b0 con 0x7f923007de20 2024-04-23T18:05:21.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f922001a8c0 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f922d7fa700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 0x7f921c03ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92300640a0 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f922004dbe0 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f922f7fe700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 0x7f921c03ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.329+0000 7f922f7fe700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 0x7f921c03ba90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9218006fd0 tx=0x7f9218006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.334+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f922004d600 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.542+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f92301a1a80 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.543+0000 7f922d7fa700 1 -- 172.21.15.32:0/1359073224 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f92301a1a80 con 0x7f923007de20 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 msgr2=0x7f921c03ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 0x7f921c03ba90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9218006fd0 tx=0x7f9218006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 msgr2=0x7f923019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9220003f40 tx=0x7f9220011890 comp rx=0 tx=0).stop 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 shutdown_connections 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f921c0395d0 0x7f921c03ba90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 --2- 172.21.15.32:0/1359073224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923007de20 0x7f923019e3f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 >> 172.21.15.32:0/1359073224 conn(0x7f92301039b0 msgr2=0x7f9230104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 shutdown_connections 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:20.545+0000 7f9236999700 1 -- 172.21.15.32:0/1359073224 wait complete. 2024-04-23T18:05:21.025 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-04-23T18:05:22.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:21 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2730066776' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T18:05:22.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setuser ceph since I am not root 2024-04-23T18:05:22.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setgroup ceph since I am not root 2024-04-23T18:05:22.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:21.904+0000 7f73f13c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:05:22.300 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:22.096+0000 7f73f13c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:05:22.300 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:22.201+0000 7f73f13c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.402+0000 7fc7fee09700 1 Processor -- start 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.402+0000 7fc7fee09700 1 -- start start 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.402+0000 7fc7fee09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.402+0000 7fc7fee09700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f807be40 con 0x7fc7f8108e30 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7fcba5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7fcba5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47726/0 (socket says 172.21.15.32:47726) 2024-04-23T18:05:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7fcba5700 1 -- 172.21.15.32:0/1258963051 learned_addr learned my addr 172.21.15.32:0/1258963051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7fcba5700 1 -- 172.21.15.32:0/1258963051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f810b760 con 0x7fc7f8108e30 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7fcba5700 1 --2- 172.21.15.32:0/1258963051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80098a0 tx=0x7fc7e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83183311c7230992 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7f77fe700 1 -- 172.21.15.32:0/1258963051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7e8004160 con 0x7fc7f8108e30 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7f77fe700 1 -- 172.21.15.32:0/1258963051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc7e8004300 con 0x7fc7f8108e30 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.403+0000 7fc7f77fe700 1 -- 172.21.15.32:0/1258963051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7e8011af0 con 0x7fc7f8108e30 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 -- 172.21.15.32:0/1258963051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 msgr2=0x7fc7f810b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 --2- 172.21.15.32:0/1258963051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80098a0 tx=0x7fc7e8009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 -- 172.21.15.32:0/1258963051 shutdown_connections 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 --2- 172.21.15.32:0/1258963051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f810b220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 -- 172.21.15.32:0/1258963051 >> 172.21.15.32:0/1258963051 conn(0x7fc7f8102820 msgr2=0x7fc7f8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 -- 172.21.15.32:0/1258963051 shutdown_connections 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 -- 172.21.15.32:0/1258963051 wait complete. 2024-04-23T18:05:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.404+0000 7fc7fee09700 1 Processor -- start 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fee09700 1 -- start start 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fee09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fee09700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f810c6d0 con 0x7fc7f8108e30 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fcba5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fcba5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47740/0 (socket says 172.21.15.32:47740) 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fcba5700 1 -- 172.21.15.32:0/2730066776 learned_addr learned my addr 172.21.15.32:0/2730066776 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fcba5700 1 -- 172.21.15.32:0/2730066776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f819c720 con 0x7fc7f8108e30 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fcba5700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8011ac0 tx=0x7fc7e8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7e8003b20 con 0x7fc7f8108e30 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc7e801a8c0 con 0x7fc7f8108e30 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7e80248d0 con 0x7fc7f8108e30 2024-04-23T18:05:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.405+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f819c9b0 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.406+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f819f6a0 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.406+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc7e8022020 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.407+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f80640a0 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.407+0000 7fc7f5ffb700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 0x7fc7e003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.407+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc7e804d490 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.407+0000 7fc7f7fff700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 0x7fc7e003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.407+0000 7fc7f7fff700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 0x7fc7e003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec006fd0 tx=0x7fc7ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.412+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7e804d1f0 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.679+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fc7f819f870 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.730+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fc7f819f870 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.732+0000 7fc7f5ffb700 1 -- 172.21.15.32:0/2730066776 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc7e802d3f0 con 0x7fc7f8108e30 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 msgr2=0x7fc7e003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 0x7fc7e003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec006fd0 tx=0x7fc7ec006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 msgr2=0x7fc7f819c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8011ac0 tx=0x7fc7e8011ba0 comp rx=0 tx=0).stop 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 shutdown_connections 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7fc7e0039670 0x7fc7e003bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 --2- 172.21.15.32:0/2730066776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8108e30 0x7fc7f819c1e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 >> 172.21.15.32:0/2730066776 conn(0x7fc7f8102820 msgr2=0x7fc7f8103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 shutdown_connections 2024-04-23T18:05:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:21.734+0000 7fc7fee09700 1 -- 172.21.15.32:0/2730066776 wait complete. 2024-04-23T18:05:22.553 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:22.534+0000 7f73f13c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:05:22.937 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:22.625+0000 7f73f13c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:05:22.937 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2730066776' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T18:05:22.937 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:22 smithi032 ceph-mon[29221]: mgrmap e5: x(active, since 6s) 2024-04-23T18:05:23.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.734+0000 7f80f68b7700 1 Processor -- start 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.734+0000 7f80f68b7700 1 -- start start 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.734+0000 7f80f68b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.734+0000 7f80f68b7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80f0107c50 con 0x7f80f0107260 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.735+0000 7f80effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:23.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.735+0000 7f80effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47766/0 (socket says 172.21.15.32:47766) 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.735+0000 7f80effff700 1 -- 172.21.15.32:0/3364383090 learned_addr learned my addr 172.21.15.32:0/3364383090 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.735+0000 7f80effff700 1 -- 172.21.15.32:0/3364383090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f0108460 con 0x7f80f0107260 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.735+0000 7f80effff700 1 --2- 172.21.15.32:0/3364383090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f80e00098a0 tx=0x7f80e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12e638eb45e19bf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.737+0000 7f80eeffd700 1 -- 172.21.15.32:0/3364383090 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e0004160 con 0x7f80f0107260 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.737+0000 7f80eeffd700 1 -- 172.21.15.32:0/3364383090 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f80e0004300 con 0x7f80f0107260 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.737+0000 7f80eeffd700 1 -- 172.21.15.32:0/3364383090 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e0011af0 con 0x7f80f0107260 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.737+0000 7f80f68b7700 1 -- 172.21.15.32:0/3364383090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 msgr2=0x7f80f0107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.737+0000 7f80f68b7700 1 --2- 172.21.15.32:0/3364383090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f80e00098a0 tx=0x7f80e0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 -- 172.21.15.32:0/3364383090 shutdown_connections 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 --2- 172.21.15.32:0/3364383090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f0107680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 -- 172.21.15.32:0/3364383090 >> 172.21.15.32:0/3364383090 conn(0x7f80f0102820 msgr2=0x7f80f0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 -- 172.21.15.32:0/3364383090 shutdown_connections 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 -- 172.21.15.32:0/3364383090 wait complete. 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 Processor -- start 2024-04-23T18:05:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.738+0000 7f80f68b7700 1 -- start start 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80f68b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80f68b7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80f007afc0 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47770/0 (socket says 172.21.15.32:47770) 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80effff700 1 -- 172.21.15.32:0/2471578127 learned_addr learned my addr 172.21.15.32:0/2471578127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80effff700 1 -- 172.21.15.32:0/2471578127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f019e970 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80effff700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e0003b20 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f80e001a8c0 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f019ec00 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.739+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e0024940 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.740+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f01a18f0 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.740+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f80e0022020 con 0x7f80f0107260 2024-04-23T18:05:23.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.741+0000 7f80ed7fa700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 0x7f80d803bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.741+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f00640a0 con 0x7f80f0107260 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.741+0000 7f80ef7fe700 1 -- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 msgr2=0x7f80d803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.741+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f80e004d480 con 0x7f80f0107260 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.741+0000 7f80ef7fe700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 0x7f80d803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.746+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80e004d1e0 con 0x7f80f0107260 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.942+0000 7f80ef7fe700 1 -- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 msgr2=0x7f80d803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:22.942+0000 7f80ef7fe700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 0x7f80d803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.023+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f80f01a1ac0 con 0x7f80f0107260 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.024+0000 7f80ed7fa700 1 -- 172.21.15.32:0/2471578127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f80e003c160 con 0x7f80f0107260 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 msgr2=0x7f80d803bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 0x7f80d803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 msgr2=0x7f80f019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e0011b70 comp rx=0 tx=0).stop 2024-04-23T18:05:23.601 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 shutdown_connections 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f80d80396c0 0x7f80d803bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 --2- 172.21.15.32:0/2471578127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0107260 0x7f80f019e430 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.029+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 >> 172.21.15.32:0/2471578127 conn(0x7f80f0102820 msgr2=0x7f80f010b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.031+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 shutdown_connections 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:23.031+0000 7f80f68b7700 1 -- 172.21.15.32:0/2471578127 wait complete. 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-23T18:05:23.602 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2024-04-23T18:05:24.000 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:23 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2471578127' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:05:24.000 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:23.881+0000 7f73f13c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:05:24.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:24.062+0000 7f73f13c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:24.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:24.238+0000 7f73f13c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:05:24.838 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:24.538+0000 7f73f13c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:05:24.838 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:24.633+0000 7f73f13c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:05:24.838 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:24.843+0000 7f73f13c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:05:25.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:25.042+0000 7f73f13c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:05:25.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:25.664+0000 7f73f13c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:05:26.403 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:26.100+0000 7f73f13c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:05:26.404 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:26.200+0000 7f73f13c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:05:26.404 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:26.408+0000 7f73f13c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:05:27.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:26.956+0000 7f73f13c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:05:27.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.051+0000 7f73f13c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:05:27.652 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.302+0000 7f73f13c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:05:27.653 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.440+0000 7f73f13c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:05:27.653 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.538+0000 7f73f13c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:05:27.653 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.657+0000 7f73f13c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:28.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:27.968+0000 7f73f13c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:05:28.863 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:28.593+0000 7f73f13c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:05:28.863 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:28.815+0000 7f73f13c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:05:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: Active manager daemon x restarted 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: Activating manager daemon x 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: mgrmap e6: x(active, starting, since 0.00344136s) 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: Manager daemon x is now available 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:28 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: Found migration_current of "None". Setting to last migration. 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:29] ENGINE Bus STARTING 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:29] ENGINE Serving on http://172.21.15.32:8765 2024-04-23T18:05:30.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:29 smithi032 ceph-mon[29221]: mgrmap e7: x(active, since 1.00571s) 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f3072302700 1 Processor -- start 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f3072302700 1 -- start start 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f3072302700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f3072302700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f306c07be60 con 0x7f306c108e30 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f306bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f306bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47778/0 (socket says 172.21.15.32:47778) 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f306bfff700 1 -- 172.21.15.32:0/1215103211 learned_addr learned my addr 172.21.15.32:0/1215103211 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f306bfff700 1 -- 172.21.15.32:0/1215103211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306c10b760 con 0x7f306c108e30 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.034+0000 7f306bfff700 1 --2- 172.21.15.32:0/1215103211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f305c0098a0 tx=0x7f305c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1914453e9f823bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f306affd700 1 -- 172.21.15.32:0/1215103211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f305c004160 con 0x7f306c108e30 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f306affd700 1 -- 172.21.15.32:0/1215103211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f305c004300 con 0x7f306c108e30 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f306affd700 1 -- 172.21.15.32:0/1215103211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f305c011af0 con 0x7f306c108e30 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f3072302700 1 -- 172.21.15.32:0/1215103211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 msgr2=0x7f306c10b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f3072302700 1 --2- 172.21.15.32:0/1215103211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f305c0098a0 tx=0x7f305c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.035+0000 7f3072302700 1 -- 172.21.15.32:0/1215103211 shutdown_connections 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 --2- 172.21.15.32:0/1215103211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c10b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 -- 172.21.15.32:0/1215103211 >> 172.21.15.32:0/1215103211 conn(0x7f306c102820 msgr2=0x7f306c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 -- 172.21.15.32:0/1215103211 shutdown_connections 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 -- 172.21.15.32:0/1215103211 wait complete. 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 Processor -- start 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.036+0000 7f3072302700 1 -- start start 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f3072302700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f3072302700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10c720 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f306bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f306bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47782/0 (socket says 172.21.15.32:47782) 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f306bfff700 1 -- 172.21.15.32:0/4109842631 learned_addr learned my addr 172.21.15.32:0/4109842631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f306bfff700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306c19c740 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f306bfff700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f305c003f40 tx=0x7f305c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f305c0038b0 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306c19c9d0 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f305c003a50 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f305c01a5c0 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.037+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306c19f6c0 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.038+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f305c01a8c0 con 0x7f306c108e30 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.039+0000 7f30697fa700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.039+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f305803c1f0 con 0x7f3058039620 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.039+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f305c04db40 con 0x7f306c108e30 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.039+0000 7f306b7fe700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.039+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.240+0000 7f306b7fe700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.240+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.641+0000 7f306b7fe700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:24.641+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:25.442+0000 7f306b7fe700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:25.442+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:27.043+0000 7f306b7fe700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3656601632 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:27.043+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:28.819+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f305c01ac00 con 0x7f306c108e30 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:28.819+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 msgr2=0x7f305803bae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:28.819+0000 7f30697fa700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.820+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f305c02d3f0 con 0x7f306c108e30 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.820+0000 7f30697fa700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 0x7f305803f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.820+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f305c029e50 con 0x7f305803cf10 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.820+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 0x7f305803f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.821+0000 7f306b7fe700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 0x7f305803f300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3054003a10 tx=0x7f3054009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.822+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f305c029e50 con 0x7f305803cf10 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.828+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f306c069b50 con 0x7f305803cf10 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.828+0000 7f30697fa700 1 -- 172.21.15.32:0/4109842631 <== mgr.14120 v2:172.21.15.32:6800/2689415742 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f306c069b50 con 0x7f305803cf10 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 msgr2=0x7f305803f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 0x7f305803f300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3054003a10 tx=0x7f3054009290 comp rx=0 tx=0).stop 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 msgr2=0x7f306c19c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f305c003f40 tx=0x7f305c011890 comp rx=0 tx=0).stop 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 shutdown_connections 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f305803cf10 0x7f305803f300 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f306c108e30 0x7f306c19c200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 --2- 172.21.15.32:0/4109842631 >> [v2:172.21.15.32:6800/3656601632,v1:172.21.15.32:6801/3656601632] conn(0x7f3058039620 0x7f305803bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 >> 172.21.15.32:0/4109842631 conn(0x7f306c102820 msgr2=0x7f306c103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 shutdown_connections 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:29.829+0000 7f3072302700 1 -- 172.21.15.32:0/4109842631 wait complete. 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-04-23T18:05:31.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:30 smithi032 ceph-mon[29221]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T18:05:31.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:30 smithi032 ceph-mon[29221]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T18:05:31.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:30 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:29] ENGINE Serving on https://172.21.15.32:7150 2024-04-23T18:05:31.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:30 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:29] ENGINE Bus STARTED 2024-04-23T18:05:31.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:30 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.857+0000 7f75cc62b700 1 Processor -- start 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.857+0000 7f75cc62b700 1 -- start start 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75cc62b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75cc62b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75c407bdc0 con 0x7f75c4108eb0 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75ca3c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75ca3c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60772/0 (socket says 172.21.15.32:60772) 2024-04-23T18:05:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75ca3c7700 1 -- 172.21.15.32:0/390774692 learned_addr learned my addr 172.21.15.32:0/390774692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75ca3c7700 1 -- 172.21.15.32:0/390774692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c410b7e0 con 0x7f75c4108eb0 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.858+0000 7f75ca3c7700 1 --2- 172.21.15.32:0/390774692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75c00098a0 tx=0x7f75c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e38a7771a1906e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75c93c5700 1 -- 172.21.15.32:0/390774692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0004160 con 0x7f75c4108eb0 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75c93c5700 1 -- 172.21.15.32:0/390774692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f75c0004300 con 0x7f75c4108eb0 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75c93c5700 1 -- 172.21.15.32:0/390774692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0011af0 con 0x7f75c4108eb0 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 -- 172.21.15.32:0/390774692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 msgr2=0x7f75c410b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 --2- 172.21.15.32:0/390774692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75c00098a0 tx=0x7f75c0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 -- 172.21.15.32:0/390774692 shutdown_connections 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 --2- 172.21.15.32:0/390774692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c410b2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 -- 172.21.15.32:0/390774692 >> 172.21.15.32:0/390774692 conn(0x7f75c4102840 msgr2=0x7f75c4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 -- 172.21.15.32:0/390774692 shutdown_connections 2024-04-23T18:05:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.859+0000 7f75cc62b700 1 -- 172.21.15.32:0/390774692 wait complete. 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75cc62b700 1 Processor -- start 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75cc62b700 1 -- start start 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75cc62b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75cc62b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75c410c6e0 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75ca3c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75ca3c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60788/0 (socket says 172.21.15.32:60788) 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.860+0000 7f75ca3c7700 1 -- 172.21.15.32:0/3313812587 learned_addr learned my addr 172.21.15.32:0/3313812587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75ca3c7700 1 -- 172.21.15.32:0/3313812587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c419c7a0 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75ca3c7700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009fb0 tx=0x7f75c0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c00036a0 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c419ca30 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c419f720 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f75c001a430 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.861+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0019550 con 0x7f75c4108eb0 2024-04-23T18:05:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.862+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f75c0019800 con 0x7f75c4108eb0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.862+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a8005360 con 0x7f75c4108eb0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.862+0000 7f75bb7fe700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 0x7f75b003ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.862+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f75c004d6c0 con 0x7f75c4108eb0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.862+0000 7f75c9bc6700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 0x7f75b003ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.863+0000 7f75c9bc6700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 0x7f75b003ba00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.868+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75c0016dd0 con 0x7f75c4108eb0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:30.873+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f75c002d960 con 0x7f75c4108eb0 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.096+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f75a8000bf0 con 0x7f75b0039540 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.101+0000 7f75bb7fe700 1 -- 172.21.15.32:0/3313812587 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f75a8000bf0 con 0x7f75b0039540 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 msgr2=0x7f75b003ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 0x7f75b003ba00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 msgr2=0x7f75c419c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009fb0 tx=0x7f75c0011940 comp rx=0 tx=0).stop 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 shutdown_connections 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f75b0039540 0x7f75b003ba00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 --2- 172.21.15.32:0/3313812587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c4108eb0 0x7f75c419c260 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 >> 172.21.15.32:0/3313812587 conn(0x7f75c4102840 msgr2=0x7f75c4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 shutdown_connections 2024-04-23T18:05:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:31.104+0000 7f75cc62b700 1 -- 172.21.15.32:0/3313812587 wait complete. 2024-04-23T18:05:32.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:31 smithi032 ceph-mon[29221]: mgrmap e8: x(active, since 2s) 2024-04-23T18:05:32.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:31 smithi032 ceph-mon[29221]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:32.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:31 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:32.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:31 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.071+0000 7fc14e7f8700 1 Processor -- start 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.071+0000 7fc14e7f8700 1 -- start start 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.071+0000 7fc14e7f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.071+0000 7fc14e7f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc14807c8c0 con 0x7fc14807de40 2024-04-23T18:05:32.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.071+0000 7fc147fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc147fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60800/0 (socket says 172.21.15.32:60800) 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc147fff700 1 -- 172.21.15.32:0/2589864909 learned_addr learned my addr 172.21.15.32:0/2589864909 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc147fff700 1 -- 172.21.15.32:0/2589864909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14807ca40 con 0x7fc14807de40 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc147fff700 1 --2- 172.21.15.32:0/2589864909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1380098a0 tx=0x7fc138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3250f70479cf6bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc146ffd700 1 -- 172.21.15.32:0/2589864909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138004160 con 0x7fc14807de40 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc146ffd700 1 -- 172.21.15.32:0/2589864909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc138004300 con 0x7fc14807de40 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.072+0000 7fc146ffd700 1 -- 172.21.15.32:0/2589864909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138011af0 con 0x7fc14807de40 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 -- 172.21.15.32:0/2589864909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 msgr2=0x7fc14807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/2589864909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1380098a0 tx=0x7fc138009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 -- 172.21.15.32:0/2589864909 shutdown_connections 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/2589864909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14807c2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 -- 172.21.15.32:0/2589864909 >> 172.21.15.32:0/2589864909 conn(0x7fc1481039b0 msgr2=0x7fc148105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:32.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 -- 172.21.15.32:0/2589864909 shutdown_connections 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 -- 172.21.15.32:0/2589864909 wait complete. 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.073+0000 7fc14e7f8700 1 Processor -- start 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc14e7f8700 1 -- start start 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc14e7f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc14e7f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc14810e900 con 0x7fc14807de40 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc147fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc147fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60814/0 (socket says 172.21.15.32:60814) 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc147fff700 1 -- 172.21.15.32:0/234673531 learned_addr learned my addr 172.21.15.32:0/234673531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.074+0000 7fc147fff700 1 -- 172.21.15.32:0/234673531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14819e8f0 con 0x7fc14807de40 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc147fff700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc138003f40 tx=0x7fc138011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1380038b0 con 0x7fc14807de40 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc138003a50 con 0x7fc14807de40 2024-04-23T18:05:32.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14819eb80 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc13801a5c0 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.075+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481a1870 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.076+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc13801a870 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.076+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc148069b50 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.076+0000 7fc1457fa700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 0x7fc13003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.076+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc13804dc00 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.076+0000 7fc1477fe700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 0x7fc13003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.077+0000 7fc1477fe700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 0x7fc13003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.081+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc13802da70 con 0x7fc14807de40 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.292+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fc148104af0 con 0x7fc130039670 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.292+0000 7fc1457fa700 1 -- 172.21.15.32:0/234673531 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fc148104af0 con 0x7fc130039670 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.294+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 msgr2=0x7fc13003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.294+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 0x7fc13003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:32.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.294+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 msgr2=0x7fc14819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.294+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc138003f40 tx=0x7fc138011890 comp rx=0 tx=0).stop 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 shutdown_connections 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc130039670 0x7fc13003bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 --2- 172.21.15.32:0/234673531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14807de40 0x7fc14819e3b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 >> 172.21.15.32:0/234673531 conn(0x7fc1481039b0 msgr2=0x7fc1481043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 shutdown_connections 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:32.295+0000 7fc14e7f8700 1 -- 172.21.15.32:0/234673531 wait complete. 2024-04-23T18:05:32.862 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-04-23T18:05:33.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:32 smithi032 ceph-mon[29221]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:34.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-mon[29221]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:34.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-mon[29221]: Generating ssh key... 2024-04-23T18:05:34.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:34.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: Generating public/private rsa key pair. 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: Your identification has been saved in /tmp/tmpuc9e95ev/key. 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: Your public key has been saved in /tmp/tmpuc9e95ev/key.pub. 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: The key fingerprint is: 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: SHA256:yddoAZtPiiF5FerAzM+giotaGtbKXxZSNEBMZmuY5es ceph-cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: The key's randomart image is: 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: +---[RSA 3072]----+ 2024-04-23T18:05:34.163 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: | +B.o +. | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: | B=o..o + | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: | o +O.+ o o | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: | ..oO + = + | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: | .o .= S = . | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: |..o . . o | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: |+..E o | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: |+=. o | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: |*o.. | 2024-04-23T18:05:34.164 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: +----[SHA256]-----+ 2024-04-23T18:05:34.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa13a626700 1 Processor -- start 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa13a626700 1 -- start start 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa13a626700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa13a626700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa134108cf0 con 0x7fa134108300 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa133fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa133fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60824/0 (socket says 172.21.15.32:60824) 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.249+0000 7fa133fff700 1 -- 172.21.15.32:0/438111067 learned_addr learned my addr 172.21.15.32:0/438111067 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa133fff700 1 -- 172.21.15.32:0/438111067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa134109500 con 0x7fa134108300 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa133fff700 1 --2- 172.21.15.32:0/438111067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1240098a0 tx=0x7fa124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc49c25eabcca162 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa132ffd700 1 -- 172.21.15.32:0/438111067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa124004160 con 0x7fa134108300 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa132ffd700 1 -- 172.21.15.32:0/438111067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa124004300 con 0x7fa134108300 2024-04-23T18:05:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa132ffd700 1 -- 172.21.15.32:0/438111067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa124011af0 con 0x7fa134108300 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa13a626700 1 -- 172.21.15.32:0/438111067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 msgr2=0x7fa134108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa13a626700 1 --2- 172.21.15.32:0/438111067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1240098a0 tx=0x7fa124009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa13a626700 1 -- 172.21.15.32:0/438111067 shutdown_connections 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.250+0000 7fa13a626700 1 --2- 172.21.15.32:0/438111067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa134108720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.251+0000 7fa13a626700 1 -- 172.21.15.32:0/438111067 >> 172.21.15.32:0/438111067 conn(0x7fa1341038e0 msgr2=0x7fa134105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.251+0000 7fa13a626700 1 -- 172.21.15.32:0/438111067 shutdown_connections 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.251+0000 7fa13a626700 1 -- 172.21.15.32:0/438111067 wait complete. 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.251+0000 7fa13a626700 1 Processor -- start 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.251+0000 7fa13a626700 1 -- start start 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa13a626700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:34.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa13a626700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa13410c8b0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa133fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa133fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60834/0 (socket says 172.21.15.32:60834) 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa133fff700 1 -- 172.21.15.32:0/3015309401 learned_addr learned my addr 172.21.15.32:0/3015309401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa133fff700 1 -- 172.21.15.32:0/3015309401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13419e960 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa133fff700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa124003f40 tx=0x7fa124011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1240038b0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa124003a50 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13419ebf0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa12401a5c0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.252+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1341a18e0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.253+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa124024430 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.253+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1340640a0 con 0x7fa134108300 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.254+0000 7fa1317fa700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 0x7fa11c03bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:34.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.254+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa12404dae0 con 0x7fa134108300 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.254+0000 7fa1337fe700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 0x7fa11c03bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.254+0000 7fa1337fe700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 0x7fa11c03bae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.259+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa12402d910 con 0x7fa134108300 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.469+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa134104a60 con 0x7fa11c039620 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.872+0000 7fa1317fa700 1 -- 172.21.15.32:0/3015309401 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa134104a60 con 0x7fa11c039620 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 msgr2=0x7fa11c03bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 0x7fa11c03bae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 msgr2=0x7fa13419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa124003f40 tx=0x7fa124011890 comp rx=0 tx=0).stop 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 shutdown_connections 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fa11c039620 0x7fa11c03bae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 --2- 172.21.15.32:0/3015309401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa134108300 0x7fa13419e420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:34.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 >> 172.21.15.32:0/3015309401 conn(0x7fa1341038e0 msgr2=0x7fa134104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:34.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 shutdown_connections 2024-04-23T18:05:34.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:33.874+0000 7fa13a626700 1 -- 172.21.15.32:0/3015309401 wait complete. 2024-04-23T18:05:35.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3cab8700 1 Processor -- start 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3cab8700 1 -- start start 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3cab8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3cab8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d38108da0 con 0x7f7d381083b0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60836/0 (socket says 172.21.15.32:60836) 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.847+0000 7f7d3659c700 1 -- 172.21.15.32:0/843594777 learned_addr learned my addr 172.21.15.32:0/843594777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.848+0000 7f7d3659c700 1 -- 172.21.15.32:0/843594777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d38109560 con 0x7f7d381083b0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.848+0000 7f7d3659c700 1 --2- 172.21.15.32:0/843594777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d280098a0 tx=0x7f7d28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702ef49374bbe874 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.848+0000 7f7d3559a700 1 -- 172.21.15.32:0/843594777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d28004160 con 0x7f7d381083b0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.848+0000 7f7d3559a700 1 -- 172.21.15.32:0/843594777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d28004300 con 0x7f7d381083b0 2024-04-23T18:05:35.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.848+0000 7f7d3559a700 1 -- 172.21.15.32:0/843594777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d28011af0 con 0x7f7d381083b0 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 -- 172.21.15.32:0/843594777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 msgr2=0x7f7d381087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/843594777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d280098a0 tx=0x7f7d28009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 -- 172.21.15.32:0/843594777 shutdown_connections 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/843594777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d381087d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 -- 172.21.15.32:0/843594777 >> 172.21.15.32:0/843594777 conn(0x7f7d381039b0 msgr2=0x7f7d38105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 -- 172.21.15.32:0/843594777 shutdown_connections 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.849+0000 7f7d3cab8700 1 -- 172.21.15.32:0/843594777 wait complete. 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3cab8700 1 Processor -- start 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3cab8700 1 -- start start 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3cab8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3cab8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3810c910 con 0x7f7d381083b0 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60838/0 (socket says 172.21.15.32:60838) 2024-04-23T18:05:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3659c700 1 -- 172.21.15.32:0/3792287687 learned_addr learned my addr 172.21.15.32:0/3792287687 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3659c700 1 -- 172.21.15.32:0/3792287687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3819c790 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.850+0000 7f7d3659c700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d28003f40 tx=0x7f7d28004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.851+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d280036a0 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.851+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3819ca20 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.851+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d28024430 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.851+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d2801a5c0 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.851+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3819f710 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7d28003840 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d380640a0 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d277fe700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 0x7f7d2003bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d2804d780 con 0x7f7d381083b0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d35d9b700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 0x7f7d2003bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:35.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.852+0000 7f7d35d9b700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 0x7f7d2003bae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006fd0 tx=0x7f7d2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:34.857+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d2800f2e0 con 0x7f7d381083b0 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.068+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f7d38104bb0 con 0x7f7d20039620 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.069+0000 7f7d277fe700 1 -- 172.21.15.32:0/3792287687 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f7d38104bb0 con 0x7f7d20039620 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 msgr2=0x7f7d2003bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 0x7f7d2003bae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006fd0 tx=0x7f7d2c006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 msgr2=0x7f7d3819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d28003f40 tx=0x7f7d28004790 comp rx=0 tx=0).stop 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 shutdown_connections 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7d20039620 0x7f7d2003bae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 --2- 172.21.15.32:0/3792287687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d381083b0 0x7f7d3819c250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 >> 172.21.15.32:0/3792287687 conn(0x7f7d381039b0 msgr2=0x7f7d381044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 shutdown_connections 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:35.071+0000 7f7d3cab8700 1 -- 172.21.15.32:0/3792287687 wait complete. 2024-04-23T18:05:35.639 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:05:35.640 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T18:05:35.640 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-04-23T18:05:35.896 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:35 smithi032 ceph-mon[29221]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:36.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:36 smithi032 ceph-mon[29221]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:38.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:37 smithi032 ceph-mon[29221]: Deploying cephadm binary to smithi032 2024-04-23T18:05:41.150 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ef281700 1 Processor -- start 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ef281700 1 -- start start 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ef281700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ef281700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f98e807c8c0 con 0x7f98e807de40 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ed01d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ed01d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60840/0 (socket says 172.21.15.32:60840) 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.101+0000 7f98ed01d700 1 -- 172.21.15.32:0/3113605314 learned_addr learned my addr 172.21.15.32:0/3113605314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ed01d700 1 -- 172.21.15.32:0/3113605314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e807ca40 con 0x7f98e807de40 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ed01d700 1 --2- 172.21.15.32:0/3113605314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98e40098a0 tx=0x7f98e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57ffa65d42308ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98dffff700 1 -- 172.21.15.32:0/3113605314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e4004160 con 0x7f98e807de40 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98dffff700 1 -- 172.21.15.32:0/3113605314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98e4004300 con 0x7f98e807de40 2024-04-23T18:05:41.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98dffff700 1 -- 172.21.15.32:0/3113605314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e4011af0 con 0x7f98e807de40 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ef281700 1 -- 172.21.15.32:0/3113605314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 msgr2=0x7f98e807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ef281700 1 --2- 172.21.15.32:0/3113605314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98e40098a0 tx=0x7f98e4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ef281700 1 -- 172.21.15.32:0/3113605314 shutdown_connections 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.102+0000 7f98ef281700 1 --2- 172.21.15.32:0/3113605314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e807c2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 -- 172.21.15.32:0/3113605314 >> 172.21.15.32:0/3113605314 conn(0x7f98e81039b0 msgr2=0x7f98e8105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 -- 172.21.15.32:0/3113605314 shutdown_connections 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 -- 172.21.15.32:0/3113605314 wait complete. 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 Processor -- start 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 -- start start 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.103+0000 7f98ef281700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f98e810e900 con 0x7f98e807de40 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ed01d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ed01d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60842/0 (socket says 172.21.15.32:60842) 2024-04-23T18:05:41.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ed01d700 1 -- 172.21.15.32:0/4116907360 learned_addr learned my addr 172.21.15.32:0/4116907360 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ed01d700 1 -- 172.21.15.32:0/4116907360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e819e920 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ed01d700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98e4003f40 tx=0x7f98e4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e40038b0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98e4003a50 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e819ebb0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e401a5c0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.104+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e81a18a0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.105+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f98e401a870 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.105+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e80640a0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.105+0000 7f98de7fc700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 0x7f98d403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.105+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f98e404dab0 con 0x7f98e807de40 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.105+0000 7f98ec81c700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 0x7f98d403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.106+0000 7f98ec81c700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 0x7f98d403bb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006fd0 tx=0x7f98d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:41.153 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.111+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98e402d3f0 con 0x7f98e807de40 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:36.329+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}) -- 0x7f98e8104b60 con 0x7f98d4039670 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.570+0000 7f98de7fc700 1 -- 172.21.15.32:0/4116907360 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f98e8104b60 con 0x7f98d4039670 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 msgr2=0x7f98d403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 0x7f98d403bb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006fd0 tx=0x7f98d8006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 msgr2=0x7f98e819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98e4003f40 tx=0x7f98e4011890 comp rx=0 tx=0).stop 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 shutdown_connections 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f98d4039670 0x7f98d403bb30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 --2- 172.21.15.32:0/4116907360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98e807de40 0x7f98e819e3e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 >> 172.21.15.32:0/4116907360 conn(0x7f98e81039b0 msgr2=0x7f98e8104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 shutdown_connections 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:40.573+0000 7f98ef281700 1 -- 172.21.15.32:0/4116907360 wait complete. 2024-04-23T18:05:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mon service... 2024-04-23T18:05:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:41 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:41 smithi032 ceph-mon[29221]: Added host smithi032 2024-04-23T18:05:41.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:41 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T18:05:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.573+0000 7f7aa784f700 1 Processor -- start 2024-04-23T18:05:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.573+0000 7f7aa784f700 1 -- start start 2024-04-23T18:05:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa784f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa784f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa007be20 con 0x7f7aa0106c20 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa55eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa55eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55454/0 (socket says 172.21.15.32:55454) 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa55eb700 1 -- 172.21.15.32:0/973274889 learned_addr learned my addr 172.21.15.32:0/973274889 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa55eb700 1 -- 172.21.15.32:0/973274889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa0107580 con 0x7f7aa0106c20 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7aa55eb700 1 --2- 172.21.15.32:0/973274889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009e80 tx=0x7f7a9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7819ae07db7efbf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7a9ffff700 1 -- 172.21.15.32:0/973274889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a90017070 con 0x7f7aa0106c20 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7a9ffff700 1 -- 172.21.15.32:0/973274889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7a9000b8c0 con 0x7f7aa0106c20 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.574+0000 7f7a9ffff700 1 -- 172.21.15.32:0/973274889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a90004490 con 0x7f7aa0106c20 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 -- 172.21.15.32:0/973274889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 msgr2=0x7f7aa0107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 --2- 172.21.15.32:0/973274889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009e80 tx=0x7f7a9000b330 comp rx=0 tx=0).stop 2024-04-23T18:05:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 -- 172.21.15.32:0/973274889 shutdown_connections 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 --2- 172.21.15.32:0/973274889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa0107040 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 -- 172.21.15.32:0/973274889 >> 172.21.15.32:0/973274889 conn(0x7f7aa0102820 msgr2=0x7f7aa0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 -- 172.21.15.32:0/973274889 shutdown_connections 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.575+0000 7f7aa784f700 1 -- 172.21.15.32:0/973274889 wait complete. 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa784f700 1 Processor -- start 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa784f700 1 -- start start 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa784f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa784f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa01085b0 con 0x7f7aa0106c20 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa55eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa55eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55456/0 (socket says 172.21.15.32:55456) 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.576+0000 7f7aa55eb700 1 -- 172.21.15.32:0/3384368857 learned_addr learned my addr 172.21.15.32:0/3384368857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7aa55eb700 1 -- 172.21.15.32:0/3384368857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa019e8d0 con 0x7f7aa0106c20 2024-04-23T18:05:42.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7aa55eb700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000b7b0 tx=0x7f7a9000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a90020020 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7a90003d10 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa019eb60 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a90017040 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.577+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa01a1850 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.578+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7a90023020 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.578+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa00640a0 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.578+0000 7f7a9e7fc700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 0x7f7a8803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.578+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7a9000f050 con 0x7f7aa0106c20 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.578+0000 7f7aa4dea700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 0x7f7a8803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.579+0000 7f7aa4dea700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 0x7f7a8803bb30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a94006fd0 tx=0x7f7a94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.583+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a9002a020 con 0x7f7aa0106c20 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.804+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f7aa002e360 con 0x7f7a88039670 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.808+0000 7f7a9e7fc700 1 -- 172.21.15.32:0/3384368857 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7aa002e360 con 0x7f7a88039670 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.810+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 msgr2=0x7f7a8803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.810+0000 7f7aa784f700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 0x7f7a8803bb30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a94006fd0 tx=0x7f7a94006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.810+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 msgr2=0x7f7aa019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.810+0000 7f7aa784f700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000b7b0 tx=0x7f7a9000be90 comp rx=0 tx=0).stop 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 shutdown_connections 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f7a88039670 0x7f7a8803bb30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 --2- 172.21.15.32:0/3384368857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa0106c20 0x7f7aa019e390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 >> 172.21.15.32:0/3384368857 conn(0x7f7aa0102820 msgr2=0x7f7aa0194fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 shutdown_connections 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:41.811+0000 7f7aa784f700 1 -- 172.21.15.32:0/3384368857 wait complete. 2024-04-23T18:05:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mgr service... 2024-04-23T18:05:42.939 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:42 smithi032 ceph-mon[29221]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:42.939 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:42 smithi032 ceph-mon[29221]: Saving service mon spec with placement count:5 2024-04-23T18:05:42.939 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:42 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:44.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T18:05:44.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f257849b700 1 Processor -- start 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f257849b700 1 -- start start 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f257849b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f257849b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f257007bea0 con 0x7f2570108dd0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f2576237700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f2576237700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55466/0 (socket says 172.21.15.32:55466) 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.343+0000 7f2576237700 1 -- 172.21.15.32:0/166965172 learned_addr learned my addr 172.21.15.32:0/166965172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.344+0000 7f2576237700 1 -- 172.21.15.32:0/166965172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257010b700 con 0x7f2570108dd0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.344+0000 7f2576237700 1 --2- 172.21.15.32:0/166965172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f256c0098a0 tx=0x7f256c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a99005a7b68c698 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.344+0000 7f2575235700 1 -- 172.21.15.32:0/166965172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f256c004160 con 0x7f2570108dd0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.344+0000 7f2575235700 1 -- 172.21.15.32:0/166965172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f256c004300 con 0x7f2570108dd0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.344+0000 7f2575235700 1 -- 172.21.15.32:0/166965172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f256c011af0 con 0x7f2570108dd0 2024-04-23T18:05:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- 172.21.15.32:0/166965172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 msgr2=0x7f257010b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 --2- 172.21.15.32:0/166965172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f256c0098a0 tx=0x7f256c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- 172.21.15.32:0/166965172 shutdown_connections 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 --2- 172.21.15.32:0/166965172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257010b1c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- 172.21.15.32:0/166965172 >> 172.21.15.32:0/166965172 conn(0x7f2570102840 msgr2=0x7f2570104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- 172.21.15.32:0/166965172 shutdown_connections 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- 172.21.15.32:0/166965172 wait complete. 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 Processor -- start 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.345+0000 7f257849b700 1 -- start start 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f257849b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f257849b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f257010c6d0 con 0x7f2570108dd0 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f2576237700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f2576237700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55476/0 (socket says 172.21.15.32:55476) 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f2576237700 1 -- 172.21.15.32:0/12744534 learned_addr learned my addr 172.21.15.32:0/12744534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:44.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f2576237700 1 -- 172.21.15.32:0/12744534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019c790 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f2576237700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f256c009fb0 tx=0x7f256c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f256c0036a0 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019ca20 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f256c01a430 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257019f710 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.346+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f256c019550 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.347+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f256c01a8e0 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.347+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25700640a0 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.347+0000 7f25677fe700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 0x7f255c03bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.347+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f256c04d8f0 con 0x7f2570108dd0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.348+0000 7f2575a36700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 0x7f255c03bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:44.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.348+0000 7f2575a36700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 0x7f255c03bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2560006fd0 tx=0x7f2560006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.352+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f256c00b3e0 con 0x7f2570108dd0 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.576+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2570103b00 con 0x7f255c039620 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.581+0000 7f25677fe700 1 -- 172.21.15.32:0/12744534 <== mgr.14120 v2:172.21.15.32:6800/2689415742 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2570103b00 con 0x7f255c039620 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 msgr2=0x7f255c03bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 0x7f255c03bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2560006fd0 tx=0x7f2560006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 msgr2=0x7f257019c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:44.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f256c009fb0 tx=0x7f256c011940 comp rx=0 tx=0).stop 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 shutdown_connections 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f255c039620 0x7f255c03bae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 --2- 172.21.15.32:0/12744534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2570108dd0 0x7f257019c250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 >> 172.21.15.32:0/12744534 conn(0x7f2570102840 msgr2=0x7f25701033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 shutdown_connections 2024-04-23T18:05:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:43.583+0000 7f257849b700 1 -- 172.21.15.32:0/12744534 wait complete. 2024-04-23T18:05:45.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:44 smithi032 ceph-mon[29221]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:45.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:44 smithi032 ceph-mon[29221]: Saving service mgr spec with placement count:2 2024-04-23T18:05:45.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:44 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:45.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:44 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:45.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:44 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:46.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.795+0000 7fe0370ff700 1 Processor -- start 2024-04-23T18:05:46.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.795+0000 7fe0370ff700 1 -- start start 2024-04-23T18:05:46.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.795+0000 7fe0370ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:46.154 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.795+0000 7fe0370ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe03007be60 con 0x7fe030108df0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe034e9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe034e9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55484/0 (socket says 172.21.15.32:55484) 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe034e9b700 1 -- 172.21.15.32:0/4124245442 learned_addr learned my addr 172.21.15.32:0/4124245442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe034e9b700 1 -- 172.21.15.32:0/4124245442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03010b720 con 0x7fe030108df0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe034e9b700 1 --2- 172.21.15.32:0/4124245442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098a0 tx=0x7fe024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a630946365fc4546 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe02f7fe700 1 -- 172.21.15.32:0/4124245442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe024004160 con 0x7fe030108df0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe02f7fe700 1 -- 172.21.15.32:0/4124245442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe024004300 con 0x7fe030108df0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.796+0000 7fe02f7fe700 1 -- 172.21.15.32:0/4124245442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe024011a20 con 0x7fe030108df0 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 -- 172.21.15.32:0/4124245442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 msgr2=0x7fe03010b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 --2- 172.21.15.32:0/4124245442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098a0 tx=0x7fe024009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 -- 172.21.15.32:0/4124245442 shutdown_connections 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 --2- 172.21.15.32:0/4124245442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03010b1e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:46.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 -- 172.21.15.32:0/4124245442 >> 172.21.15.32:0/4124245442 conn(0x7fe030102820 msgr2=0x7fe030104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 -- 172.21.15.32:0/4124245442 shutdown_connections 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.797+0000 7fe0370ff700 1 -- 172.21.15.32:0/4124245442 wait complete. 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe0370ff700 1 Processor -- start 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe0370ff700 1 -- start start 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe0370ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe0370ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe03010c720 con 0x7fe030108df0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe034e9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe034e9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55494/0 (socket says 172.21.15.32:55494) 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe034e9b700 1 -- 172.21.15.32:0/2128985095 learned_addr learned my addr 172.21.15.32:0/2128985095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.798+0000 7fe034e9b700 1 -- 172.21.15.32:0/2128985095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03019e900 con 0x7fe030108df0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe034e9b700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0240116e0 tx=0x7fe024011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe024003a60 con 0x7fe030108df0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe02401b8f0 con 0x7fe030108df0 2024-04-23T18:05:46.156 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03019eb90 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe024019860 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.799+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0301a1880 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.800+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe024019b10 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.800+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0300640a0 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.800+0000 7fe02dffb700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 0x7fe02003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.800+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe02404df10 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.800+0000 7fe02ffff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 0x7fe02003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.801+0000 7fe02ffff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 0x7fe02003bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe01c006fd0 tx=0x7fe01c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:44.805+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe02401f030 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.035+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fe0301a1e50 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.038+0000 7fe02dffb700 1 -- 172.21.15.32:0/2128985095 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fe024029260 con 0x7fe030108df0 2024-04-23T18:05:46.157 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 msgr2=0x7fe02003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 0x7fe02003bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe01c006fd0 tx=0x7fe01c006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 msgr2=0x7fe03019e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0240116e0 tx=0x7fe024011aa0 comp rx=0 tx=0).stop 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 shutdown_connections 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe020039670 0x7fe02003bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 --2- 172.21.15.32:0/2128985095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe030108df0 0x7fe03019e3c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 >> 172.21.15.32:0/2128985095 conn(0x7fe030102820 msgr2=0x7fe0301031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 shutdown_connections 2024-04-23T18:05:46.158 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:45.041+0000 7fe0370ff700 1 -- 172.21.15.32:0/2128985095 wait complete. 2024-04-23T18:05:46.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:46 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2128985095' entity='client.admin' 2024-04-23T18:05:46.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:46 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:47.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.590+0000 7fc44032a700 1 Processor -- start 2024-04-23T18:05:47.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.590+0000 7fc44032a700 1 -- start start 2024-04-23T18:05:47.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc44032a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:47.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc44032a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc438107c50 con 0x7fc438107260 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc43e0c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc43e0c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55510/0 (socket says 172.21.15.32:55510) 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc43e0c6700 1 -- 172.21.15.32:0/942615044 learned_addr learned my addr 172.21.15.32:0/942615044 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.591+0000 7fc43e0c6700 1 -- 172.21.15.32:0/942615044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc438108460 con 0x7fc438107260 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc43e0c6700 1 --2- 172.21.15.32:0/942615044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc434009e80 tx=0x7fc43400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dec9ab7199cbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc43d0c4700 1 -- 172.21.15.32:0/942615044 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434017070 con 0x7fc438107260 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc43d0c4700 1 -- 172.21.15.32:0/942615044 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc43400b8c0 con 0x7fc438107260 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc43d0c4700 1 -- 172.21.15.32:0/942615044 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434004490 con 0x7fc438107260 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc44032a700 1 -- 172.21.15.32:0/942615044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 msgr2=0x7fc438107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.592+0000 7fc44032a700 1 --2- 172.21.15.32:0/942615044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc434009e80 tx=0x7fc43400b330 comp rx=0 tx=0).stop 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- 172.21.15.32:0/942615044 shutdown_connections 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 --2- 172.21.15.32:0/942615044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc438107680 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- 172.21.15.32:0/942615044 >> 172.21.15.32:0/942615044 conn(0x7fc438102840 msgr2=0x7fc438104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- 172.21.15.32:0/942615044 shutdown_connections 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- 172.21.15.32:0/942615044 wait complete. 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 Processor -- start 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- start start 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.593+0000 7fc44032a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc43810b810 con 0x7fc438107260 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc43e0c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc43e0c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55520/0 (socket says 172.21.15.32:55520) 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc43e0c6700 1 -- 172.21.15.32:0/3111770194 learned_addr learned my addr 172.21.15.32:0/3111770194 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc43e0c6700 1 -- 172.21.15.32:0/3111770194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43819cdd0 con 0x7fc438107260 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc43e0c6700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc43400bca0 tx=0x7fc43400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434020020 con 0x7fc438107260 2024-04-23T18:05:47.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc434003b50 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4381a13e0 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434017040 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.594+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4381a1800 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.595+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc41c005360 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.595+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc434023020 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.596+0000 7fc42f7fe700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 0x7fc42403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.596+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc43400f050 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.596+0000 7fc43d8c5700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 0x7fc42403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.596+0000 7fc43d8c5700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 0x7fc42403bb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc428006fd0 tx=0x7fc428006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.601+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc43402a020 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.813+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fc41c005650 con 0x7fc438107260 2024-04-23T18:05:47.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.816+0000 7fc42f7fe700 1 -- 172.21.15.32:0/3111770194 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fc434003d00 con 0x7fc438107260 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 msgr2=0x7fc42403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 0x7fc42403bb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc428006fd0 tx=0x7fc428006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 msgr2=0x7fc43807ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc43400bca0 tx=0x7fc43400bd80 comp rx=0 tx=0).stop 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 shutdown_connections 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc4240396c0 0x7fc42403bb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 --2- 172.21.15.32:0/3111770194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc438107260 0x7fc43807ba40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 >> 172.21.15.32:0/3111770194 conn(0x7fc438102840 msgr2=0x7fc438103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.818+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 shutdown_connections 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:46.819+0000 7fc44032a700 1 -- 172.21.15.32:0/3111770194 wait complete. 2024-04-23T18:05:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-04-23T18:05:48.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:47 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3111770194' entity='client.admin' 2024-04-23T18:05:48.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:47 smithi032 ceph-mon[29221]: from='mgr.14120 172.21.15.32:0/528173944' entity='mgr.x' 2024-04-23T18:05:49.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:48 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/44920823' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T18:05:49.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:48 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setuser ceph since I am not root 2024-04-23T18:05:49.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:48 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setgroup ceph since I am not root 2024-04-23T18:05:49.009 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:49.007+0000 7f7a5d0e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:05:49.406 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:49.201+0000 7f7a5d0e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:05:49.406 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:49.307+0000 7f7a5d0e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:05:49.409 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.835+0000 7fe73b5b0700 1 Processor -- start 2024-04-23T18:05:49.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.835+0000 7fe73b5b0700 1 -- start start 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.835+0000 7fe73b5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.835+0000 7fe73b5b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe734107090 con 0x7fe7341066a0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe73934c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe73934c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47470/0 (socket says 172.21.15.32:47470) 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe73934c700 1 -- 172.21.15.32:0/4234120629 learned_addr learned my addr 172.21.15.32:0/4234120629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe73934c700 1 -- 172.21.15.32:0/4234120629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe734109640 con 0x7fe7341066a0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe73934c700 1 --2- 172.21.15.32:0/4234120629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7300098a0 tx=0x7fe730009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d279bea623c3765e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe72bfff700 1 -- 172.21.15.32:0/4234120629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe730004160 con 0x7fe7341066a0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe72bfff700 1 -- 172.21.15.32:0/4234120629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe730004300 con 0x7fe7341066a0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.836+0000 7fe72bfff700 1 -- 172.21.15.32:0/4234120629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe730011a20 con 0x7fe7341066a0 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 -- 172.21.15.32:0/4234120629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 msgr2=0x7fe734106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/4234120629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7300098a0 tx=0x7fe730009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 -- 172.21.15.32:0/4234120629 shutdown_connections 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/4234120629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734106ac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 -- 172.21.15.32:0/4234120629 >> 172.21.15.32:0/4234120629 conn(0x7fe734101fa0 msgr2=0x7fe7341043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 -- 172.21.15.32:0/4234120629 shutdown_connections 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.837+0000 7fe73b5b0700 1 -- 172.21.15.32:0/4234120629 wait complete. 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73b5b0700 1 Processor -- start 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73b5b0700 1 -- start start 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73b5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73b5b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe73410c9f0 con 0x7fe7341066a0 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73934c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73934c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47472/0 (socket says 172.21.15.32:47472) 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.838+0000 7fe73934c700 1 -- 172.21.15.32:0/44920823 learned_addr learned my addr 172.21.15.32:0/44920823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe73934c700 1 -- 172.21.15.32:0/44920823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe734109400 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe73934c700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe730000c00 tx=0x7fe730011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe730003aa0 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7341075f0 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe734107b30 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73001b8f0 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.839+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe730023870 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.840+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe718005360 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.840+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe730023b20 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.840+0000 7fe72a7fc700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 0x7fe72003bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.841+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe73004de50 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.841+0000 7fe738b4b700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 0x7fe72003bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.841+0000 7fe738b4b700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 0x7fe72003bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe724006fd0 tx=0x7fe724006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:47.849+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe73001f030 con 0x7fe7341066a0 2024-04-23T18:05:49.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.125+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fe718005650 con 0x7fe7341066a0 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.819+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fe73004f070 con 0x7fe7341066a0 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.821+0000 7fe72a7fc700 1 -- 172.21.15.32:0/44920823 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fe73002c720 con 0x7fe7341066a0 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 msgr2=0x7fe72003bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 0x7fe72003bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe724006fd0 tx=0x7fe724006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 msgr2=0x7fe734108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe730000c00 tx=0x7fe730011aa0 comp rx=0 tx=0).stop 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 shutdown_connections 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fe7200396c0 0x7fe72003bb80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 --2- 172.21.15.32:0/44920823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7341066a0 0x7fe734108ec0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 >> 172.21.15.32:0/44920823 conn(0x7fe734101fa0 msgr2=0x7fe7341029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 shutdown_connections 2024-04-23T18:05:49.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:48.823+0000 7fe73b5b0700 1 -- 172.21.15.32:0/44920823 wait complete. 2024-04-23T18:05:49.718 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:49.633+0000 7f7a5d0e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:05:49.719 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:49.723+0000 7f7a5d0e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:05:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/44920823' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T18:05:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:49 smithi032 ceph-mon[29221]: mgrmap e9: x(active, since 20s) 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.843+0000 7f91c0051700 1 Processor -- start 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.843+0000 7f91c0051700 1 -- start start 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91c0051700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91c0051700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91b807c8c0 con 0x7f91b807de90 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91bdded700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91bdded700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47508/0 (socket says 172.21.15.32:47508) 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91bdded700 1 -- 172.21.15.32:0/4254652446 learned_addr learned my addr 172.21.15.32:0/4254652446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91bdded700 1 -- 172.21.15.32:0/4254652446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b807ca40 con 0x7f91b807de90 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.844+0000 7f91bdded700 1 --2- 172.21.15.32:0/4254652446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91b40098a0 tx=0x7f91b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80836080e05df602 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91bcdeb700 1 -- 172.21.15.32:0/4254652446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b4004160 con 0x7f91b807de90 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91bcdeb700 1 -- 172.21.15.32:0/4254652446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b4004300 con 0x7f91b807de90 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91bcdeb700 1 -- 172.21.15.32:0/4254652446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b4011a20 con 0x7f91b807de90 2024-04-23T18:05:50.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 -- 172.21.15.32:0/4254652446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 msgr2=0x7f91b807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 --2- 172.21.15.32:0/4254652446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91b40098a0 tx=0x7f91b4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 -- 172.21.15.32:0/4254652446 shutdown_connections 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 --2- 172.21.15.32:0/4254652446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b807c2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 -- 172.21.15.32:0/4254652446 >> 172.21.15.32:0/4254652446 conn(0x7f91b81038c0 msgr2=0x7f91b8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 -- 172.21.15.32:0/4254652446 shutdown_connections 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.845+0000 7f91c0051700 1 -- 172.21.15.32:0/4254652446 wait complete. 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91c0051700 1 Processor -- start 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91c0051700 1 -- start start 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91c0051700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91c0051700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91b810e940 con 0x7f91b807de90 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91bdded700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91bdded700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47520/0 (socket says 172.21.15.32:47520) 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.846+0000 7f91bdded700 1 -- 172.21.15.32:0/2096067880 learned_addr learned my addr 172.21.15.32:0/2096067880 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91bdded700 1 -- 172.21.15.32:0/2096067880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b819e8f0 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91bdded700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91b40116e0 tx=0x7f91b4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b40039e0 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b401b8f0 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b4023760 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b819eb80 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.847+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b81a1870 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f91b4023920 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b80640a0 con 0x7f91b807de90 2024-04-23T18:05:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91aaffd700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 0x7f91a403bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f91b404e0c0 con 0x7f91b807de90 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91bd5ec700 1 -- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 msgr2=0x7f91a403bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.848+0000 7f91bd5ec700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 0x7f91a403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:49.853+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91b401f030 con 0x7f91b807de90 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.049+0000 7f91bd5ec700 1 -- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 msgr2=0x7f91a403bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.049+0000 7f91bd5ec700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 0x7f91a403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.163+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f91b81a1ae0 con 0x7f91b807de90 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.164+0000 7f91aaffd700 1 -- 172.21.15.32:0/2096067880 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f91b4018e30 con 0x7f91b807de90 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.168+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 msgr2=0x7f91a403bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.168+0000 7f91c0051700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 0x7f91a403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.168+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 msgr2=0x7f91b819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.168+0000 7f91c0051700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91b40116e0 tx=0x7f91b4011aa0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 shutdown_connections 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7f91a40396d0 0x7f91a403bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 --2- 172.21.15.32:0/2096067880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b807de90 0x7f91b819e3b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 >> 172.21.15.32:0/2096067880 conn(0x7f91b81038c0 msgr2=0x7f91b8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 shutdown_connections 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:50.169+0000 7f91c0051700 1 -- 172.21.15.32:0/2096067880 wait complete. 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-23T18:05:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-04-23T18:05:51.098 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:50 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2096067880' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:05:51.098 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.015+0000 7f7a5d0e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:05:51.369 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.197+0000 7f7a5d0e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:51.369 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.373+0000 7f7a5d0e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:05:51.980 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.680+0000 7f7a5d0e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:05:51.980 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.775+0000 7f7a5d0e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:05:51.980 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:51.985+0000 7f7a5d0e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:05:52.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:52.186+0000 7f7a5d0e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:05:53.244 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:52.809+0000 7f7a5d0e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:05:53.508 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:53.249+0000 7f7a5d0e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:05:53.508 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:53.349+0000 7f7a5d0e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:05:54.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:53.558+0000 7f7a5d0e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:05:54.448 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.103+0000 7f7a5d0e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:05:54.448 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.198+0000 7f7a5d0e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:05:54.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.453+0000 7f7a5d0e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:05:54.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.593+0000 7f7a5d0e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:05:54.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.693+0000 7f7a5d0e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:05:55.122 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:54.814+0000 7f7a5d0e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:05:56.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:55.127+0000 7f7a5d0e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:05:56.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:55.757+0000 7f7a5d0e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:05:56.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:05:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:05:55.981+0000 7f7a5d0e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:05:56.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: Active manager daemon x restarted 2024-04-23T18:05:56.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: Activating manager daemon x 2024-04-23T18:05:56.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: mgrmap e10: x(active, starting, since 0.00347887s) 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: Manager daemon x is now available 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:05:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:05:57.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:05:57.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:56] ENGINE Bus STARTING 2024-04-23T18:05:57.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:56] ENGINE Serving on https://172.21.15.32:7150 2024-04-23T18:05:57.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:56] ENGINE Serving on http://172.21.15.32:8765 2024-04-23T18:05:57.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: [23/Apr/2024:18:05:56] ENGINE Bus STARTED 2024-04-23T18:05:57.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:57 smithi032 ceph-mon[29221]: mgrmap e11: x(active, since 1.00666s) 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.131+0000 7fc391cb2700 1 Processor -- start 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.131+0000 7fc391cb2700 1 -- start start 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.131+0000 7fc391cb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.131+0000 7fc391cb2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc38c07bdc0 con 0x7fc38c108ea0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47534/0 (socket says 172.21.15.32:47534) 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38b7fe700 1 -- 172.21.15.32:0/3866621623 learned_addr learned my addr 172.21.15.32:0/3866621623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38b7fe700 1 -- 172.21.15.32:0/3866621623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38c10b7d0 con 0x7fc38c108ea0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38b7fe700 1 --2- 172.21.15.32:0/3866621623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3800098a0 tx=0x7fc380009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f747a7cba597b4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38a7fc700 1 -- 172.21.15.32:0/3866621623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc380004160 con 0x7fc38c108ea0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38a7fc700 1 -- 172.21.15.32:0/3866621623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc380004300 con 0x7fc38c108ea0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.132+0000 7fc38a7fc700 1 -- 172.21.15.32:0/3866621623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc380011a20 con 0x7fc38c108ea0 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 -- 172.21.15.32:0/3866621623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 msgr2=0x7fc38c10b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 --2- 172.21.15.32:0/3866621623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3800098a0 tx=0x7fc380009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 -- 172.21.15.32:0/3866621623 shutdown_connections 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 --2- 172.21.15.32:0/3866621623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c10b290 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 -- 172.21.15.32:0/3866621623 >> 172.21.15.32:0/3866621623 conn(0x7fc38c102890 msgr2=0x7fc38c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 -- 172.21.15.32:0/3866621623 shutdown_connections 2024-04-23T18:05:57.899 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.133+0000 7fc391cb2700 1 -- 172.21.15.32:0/3866621623 wait complete. 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc391cb2700 1 Processor -- start 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc391cb2700 1 -- start start 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc391cb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc391cb2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc38c10c710 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc38b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc38b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47546/0 (socket says 172.21.15.32:47546) 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.134+0000 7fc38b7fe700 1 -- 172.21.15.32:0/4121273391 learned_addr learned my addr 172.21.15.32:0/4121273391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc38b7fe700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38c19c6a0 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc38b7fe700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc380000c00 tx=0x7fc380011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc380003aa0 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc38001b8f0 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38c19c930 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc38c19f620 con 0x7fc38c108ea0 2024-04-23T18:05:57.900 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.135+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc380023870 con 0x7fc38c108ea0 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fc380023b20 con 0x7fc38c108ea0 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc388ff9700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc37403c2f0 con 0x7fc374039720 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc38004de40 con 0x7fc38c108ea0 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc38affd700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.136+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.337+0000 7fc38affd700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.337+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.738+0000 7fc38affd700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:51.738+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:52.539+0000 7fc38affd700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:52.539+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:54.140+0000 7fc38affd700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/2689415742 2024-04-23T18:05:57.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:54.140+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:55.985+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fc38002c720 con 0x7fc38c108ea0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:55.985+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 msgr2=0x7fc37403bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:55.985+0000 7fc388ff9700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.986+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fc3800500f0 con 0x7fc38c108ea0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.987+0000 7fc388ff9700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 0x7fc37403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.987+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc37403c2f0 con 0x7fc37403cfc0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.987+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 0x7fc37403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.988+0000 7fc38affd700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 0x7fc37403f3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc37c003a10 tx=0x7fc37c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.988+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fc37403c2f0 con 0x7fc37403cfc0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc378002630 con 0x7fc37403cfc0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc388ff9700 1 -- 172.21.15.32:0/4121273391 <== mgr.14152 v2:172.21.15.32:6800/3652066649 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fc378002630 con 0x7fc37403cfc0 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 msgr2=0x7fc37403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc391cb2700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 0x7fc37403f3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc37c003a10 tx=0x7fc37c009270 comp rx=0 tx=0).stop 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 msgr2=0x7fc38c19c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.995+0000 7fc391cb2700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc380000c00 tx=0x7fc380011aa0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 shutdown_connections 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc37403cfc0 0x7fc37403f3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38c108ea0 0x7fc38c19c160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 --2- 172.21.15.32:0/4121273391 >> [v2:172.21.15.32:6800/2689415742,v1:172.21.15.32:6801/2689415742] conn(0x7fc374039720 0x7fc37403bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 >> 172.21.15.32:0/4121273391 conn(0x7fc38c102890 msgr2=0x7fc38c103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 shutdown_connections 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:56.996+0000 7fc391cb2700 1 -- 172.21.15.32:0/4121273391 wait complete. 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-04-23T18:05:57.903 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-04-23T18:05:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:58 smithi032 ceph-mon[29221]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T18:05:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:58 smithi032 ceph-mon[29221]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T18:05:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.351+0000 7f1e7223e700 1 Processor -- start 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.351+0000 7f1e7223e700 1 -- start start 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.351+0000 7f1e7223e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.351+0000 7f1e7223e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c107c30 con 0x7f1e6c107240 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51612/0 (socket says 172.21.15.32:51612) 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6b7fe700 1 -- 172.21.15.32:0/1193443663 learned_addr learned my addr 172.21.15.32:0/1193443663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6b7fe700 1 -- 172.21.15.32:0/1193443663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c108440 con 0x7f1e6c107240 2024-04-23T18:05:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6b7fe700 1 --2- 172.21.15.32:0/1193443663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0098a0 tx=0x7f1e5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796a8d249f08807c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6a7fc700 1 -- 172.21.15.32:0/1193443663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e5c004160 con 0x7f1e6c107240 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6a7fc700 1 -- 172.21.15.32:0/1193443663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c004300 con 0x7f1e6c107240 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.352+0000 7f1e6a7fc700 1 -- 172.21.15.32:0/1193443663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e5c011a20 con 0x7f1e6c107240 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 -- 172.21.15.32:0/1193443663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 msgr2=0x7f1e6c107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1193443663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0098a0 tx=0x7f1e5c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 -- 172.21.15.32:0/1193443663 shutdown_connections 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1193443663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c107660 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 -- 172.21.15.32:0/1193443663 >> 172.21.15.32:0/1193443663 conn(0x7f1e6c102820 msgr2=0x7f1e6c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 -- 172.21.15.32:0/1193443663 shutdown_connections 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.353+0000 7f1e7223e700 1 -- 172.21.15.32:0/1193443663 wait complete. 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e7223e700 1 Processor -- start 2024-04-23T18:05:59.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e7223e700 1 -- start start 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e7223e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e7223e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c10b7f0 con 0x7f1e6c107240 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51618/0 (socket says 172.21.15.32:51618) 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.354+0000 7f1e6b7fe700 1 -- 172.21.15.32:0/1600451202 learned_addr learned my addr 172.21.15.32:0/1600451202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e6b7fe700 1 -- 172.21.15.32:0/1600451202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c19ef30 con 0x7f1e6c107240 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e6b7fe700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0116e0 tx=0x7f1e5c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e5c0039e0 con 0x7f1e6c107240 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c01b8f0 con 0x7f1e6c107240 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6c19f100 con 0x7f1e6c107240 2024-04-23T18:05:59.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e5c023760 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.355+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6c19f4a0 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.356+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f1e5c023a10 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.356+0000 7f1e68ff9700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 0x7f1e5803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.356+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6c0640a0 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.356+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1e5c04de30 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.356+0000 7f1e6affd700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 0x7f1e5803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.357+0000 7f1e6affd700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 0x7f1e5803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e54006fd0 tx=0x7f1e54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.362+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e5c01f030 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.576+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f1e6c1039a0 con 0x7f1e58039560 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.595+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f1e5c02c910 con 0x7f1e6c107240 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.697+0000 7f1e68ff9700 1 -- 172.21.15.32:0/1600451202 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f1e6c1039a0 con 0x7f1e58039560 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 msgr2=0x7f1e5803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:59.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 0x7f1e5803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e54006fd0 tx=0x7f1e54006e40 comp rx=0 tx=0).stop 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 msgr2=0x7f1e6c07ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0116e0 tx=0x7f1e5c011aa0 comp rx=0 tx=0).stop 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 shutdown_connections 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1e58039560 0x7f1e5803ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 --2- 172.21.15.32:0/1600451202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e6c107240 0x7f1e6c07ba40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 >> 172.21.15.32:0/1600451202 conn(0x7f1e6c102820 msgr2=0x7f1e6c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 shutdown_connections 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:58.699+0000 7f1e7223e700 1 -- 172.21.15.32:0/1600451202 wait complete. 2024-04-23T18:05:59.231 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-04-23T18:05:59.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:59 smithi032 ceph-mon[29221]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:05:59.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:59 smithi032 ceph-mon[29221]: mgrmap e12: x(active, since 2s) 2024-04-23T18:05:59.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:05:59.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:05:59.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:05:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PEPkZSY.ajPX7MxVFOiUneqQeVYWlGQaBHpSoICpihPgSseFecjAK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713895560, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff4a612a700 1 Processor -- start 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff4a612a700 1 -- start start 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff4a612a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff4a612a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a0108da0 con 0x7ff4a01083b0 2024-04-23T18:06:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff49f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff49f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51630/0 (socket says 172.21.15.32:51630) 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff49f7fe700 1 -- 172.21.15.32:0/1632798982 learned_addr learned my addr 172.21.15.32:0/1632798982 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff49f7fe700 1 -- 172.21.15.32:0/1632798982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a0109560 con 0x7ff4a01083b0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.666+0000 7ff49f7fe700 1 --2- 172.21.15.32:0/1632798982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098a0 tx=0x7ff490009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5ec61d1cc8da68 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff49e7fc700 1 -- 172.21.15.32:0/1632798982 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff490004160 con 0x7ff4a01083b0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff49e7fc700 1 -- 172.21.15.32:0/1632798982 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff490004300 con 0x7ff4a01083b0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff49e7fc700 1 -- 172.21.15.32:0/1632798982 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff490011a20 con 0x7ff4a01083b0 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 -- 172.21.15.32:0/1632798982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 msgr2=0x7ff4a01087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 --2- 172.21.15.32:0/1632798982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098a0 tx=0x7ff490009bb0 comp rx=0 tx=0).stop 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 -- 172.21.15.32:0/1632798982 shutdown_connections 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 --2- 172.21.15.32:0/1632798982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a01087d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 -- 172.21.15.32:0/1632798982 >> 172.21.15.32:0/1632798982 conn(0x7ff4a01039b0 msgr2=0x7ff4a0105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:00.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 -- 172.21.15.32:0/1632798982 shutdown_connections 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.667+0000 7ff4a612a700 1 -- 172.21.15.32:0/1632798982 wait complete. 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff4a612a700 1 Processor -- start 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff4a612a700 1 -- start start 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff4a612a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff4a612a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010c910 con 0x7ff4a01083b0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff49f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff49f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51640/0 (socket says 172.21.15.32:51640) 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.668+0000 7ff49f7fe700 1 -- 172.21.15.32:0/2781759595 learned_addr learned my addr 172.21.15.32:0/2781759595 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff49f7fe700 1 -- 172.21.15.32:0/2781759595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a019c6e0 con 0x7ff4a01083b0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff49f7fe700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff4900116e0 tx=0x7ff490011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4900039e0 con 0x7ff4a01083b0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff49001b8f0 con 0x7ff4a01083b0 2024-04-23T18:06:00.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a019c970 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff490023760 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.669+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a019f660 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.670+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff490023900 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.670+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a00640a0 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.670+0000 7ff49cff9700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 0x7ff48803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.670+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff49004e140 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.670+0000 7ff49effd700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 0x7ff48803bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.671+0000 7ff49effd700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 0x7ff48803bb90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff494006fd0 tx=0x7ff494006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.676+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff49001f030 con 0x7ff4a01083b0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:05:59.918+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff4a019ce50 con 0x7ff4880396d0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.165+0000 7ff49cff9700 1 -- 172.21.15.32:0/2781759595 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff4a019ce50 con 0x7ff4880396d0 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 msgr2=0x7ff48803bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:00.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 0x7ff48803bb90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff494006fd0 tx=0x7ff494006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 msgr2=0x7ff4a019c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff4900116e0 tx=0x7ff490011aa0 comp rx=0 tx=0).stop 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 shutdown_connections 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4880396d0 0x7ff48803bb90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 --2- 172.21.15.32:0/2781759595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4a01083b0 0x7ff4a019c1a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 >> 172.21.15.32:0/2781759595 conn(0x7ff4a01039b0 msgr2=0x7ff4a0104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 shutdown_connections 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:00.167+0000 7ff4a612a700 1 -- 172.21.15.32:0/2781759595 wait complete. 2024-04-23T18:06:00.833 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-04-23T18:06:01.481 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:01 smithi032 ceph-mon[29221]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:01.481 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:02.480 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2283434132' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T18:06:02.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T18:06:02.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99f26c0700 1 Processor -- start 2024-04-23T18:06:02.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99f26c0700 1 -- start start 2024-04-23T18:06:02.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99f26c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99f26c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec107030 con 0x7f99ec106640 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51654/0 (socket says 172.21.15.32:51654) 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99ebfff700 1 -- 172.21.15.32:0/3610576704 learned_addr learned my addr 172.21.15.32:0/3610576704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.743+0000 7f99ebfff700 1 -- 172.21.15.32:0/3610576704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec1095d0 con 0x7f99ec106640 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99ebfff700 1 --2- 172.21.15.32:0/3610576704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0098a0 tx=0x7f99dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1498b4b818d3d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99eaffd700 1 -- 172.21.15.32:0/3610576704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99dc004160 con 0x7f99ec106640 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99eaffd700 1 -- 172.21.15.32:0/3610576704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99dc004300 con 0x7f99ec106640 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99eaffd700 1 -- 172.21.15.32:0/3610576704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99dc011a20 con 0x7f99ec106640 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99f26c0700 1 -- 172.21.15.32:0/3610576704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 msgr2=0x7f99ec106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.744+0000 7f99f26c0700 1 --2- 172.21.15.32:0/3610576704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0098a0 tx=0x7f99dc009bb0 comp rx=0 tx=0).stop 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 -- 172.21.15.32:0/3610576704 shutdown_connections 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 --2- 172.21.15.32:0/3610576704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec106a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 -- 172.21.15.32:0/3610576704 >> 172.21.15.32:0/3610576704 conn(0x7f99ec101fb0 msgr2=0x7f99ec1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 -- 172.21.15.32:0/3610576704 shutdown_connections 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 -- 172.21.15.32:0/3610576704 wait complete. 2024-04-23T18:06:02.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 Processor -- start 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.745+0000 7f99f26c0700 1 -- start start 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99f26c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99f26c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c980 con 0x7f99ec106640 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51662/0 (socket says 172.21.15.32:51662) 2024-04-23T18:06:02.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99ebfff700 1 -- 172.21.15.32:0/2283434132 learned_addr learned my addr 172.21.15.32:0/2283434132 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99ebfff700 1 -- 172.21.15.32:0/2283434132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec19c6b0 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99ebfff700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0116e0 tx=0x7f99dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99dc0039e0 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99dc01b8f0 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec19c940 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99dc023760 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.746+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec19f630 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.747+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f99dc0239c0 con 0x7f99ec106640 2024-04-23T18:06:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.747+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec0640a0 con 0x7f99ec106640 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.748+0000 7f99e97fa700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 0x7f99d403bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.748+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f99dc04dfb0 con 0x7f99ec106640 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.748+0000 7f99eb7fe700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 0x7f99d403bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.748+0000 7f99eb7fe700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 0x7f99d403bb90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.753+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99dc01f030 con 0x7f99ec106640 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.959+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f99ec19fbe0 con 0x7f99ec106640 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.960+0000 7f99e97fa700 1 -- 172.21.15.32:0/2283434132 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f99dc016260 con 0x7f99ec106640 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 msgr2=0x7f99d403bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 0x7f99d403bb90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 msgr2=0x7f99ec19c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0116e0 tx=0x7f99dc011aa0 comp rx=0 tx=0).stop 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 shutdown_connections 2024-04-23T18:06:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f99d40396d0 0x7f99d403bb90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 --2- 172.21.15.32:0/2283434132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99ec106640 0x7f99ec19c170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 >> 172.21.15.32:0/2283434132 conn(0x7f99ec101fb0 msgr2=0x7f99ec1028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 shutdown_connections 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:01.962+0000 7f99f26c0700 1 -- 172.21.15.32:0/2283434132 wait complete. 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:06:02.496 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-23T18:06:02.503 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:06:02.503 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-23T18:06:02.503 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-23T18:06:02.503 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032:8443/ 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: Password: hc50zutshq 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:02.505 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config directory 2024-04-23T18:06:02.506 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2024-04-23T18:06:03.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:03 smithi032 ceph-mon[29221]: mgrmap e13: x(active, since 6s) 2024-04-23T18:06:03.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.034+0000 7f55f0450700 1 Processor -- start 2024-04-23T18:06:03.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.034+0000 7f55f0450700 1 -- start start 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55f0450700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55f0450700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f55e807be60 con 0x7f55e8108e30 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55ee1ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55ee1ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51666/0 (socket says 172.21.15.32:51666) 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55ee1ec700 1 -- 172.21.15.32:0/3137517170 learned_addr learned my addr 172.21.15.32:0/3137517170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.035+0000 7f55ee1ec700 1 -- 172.21.15.32:0/3137517170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e810b760 con 0x7f55e8108e30 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55ee1ec700 1 --2- 172.21.15.32:0/3137517170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa91c0b7e3eca7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55ed1ea700 1 -- 172.21.15.32:0/3137517170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e4004160 con 0x7f55e8108e30 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55ed1ea700 1 -- 172.21.15.32:0/3137517170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e4004300 con 0x7f55e8108e30 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55ed1ea700 1 -- 172.21.15.32:0/3137517170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e4011a20 con 0x7f55e8108e30 2024-04-23T18:06:03.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55f0450700 1 -- 172.21.15.32:0/3137517170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 msgr2=0x7f55e810b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.036+0000 7f55f0450700 1 --2- 172.21.15.32:0/3137517170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- 172.21.15.32:0/3137517170 shutdown_connections 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 --2- 172.21.15.32:0/3137517170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e810b220 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- 172.21.15.32:0/3137517170 >> 172.21.15.32:0/3137517170 conn(0x7f55e8102860 msgr2=0x7f55e8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- 172.21.15.32:0/3137517170 shutdown_connections 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- 172.21.15.32:0/3137517170 wait complete. 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 Processor -- start 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- start start 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.037+0000 7f55f0450700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f55e810c760 con 0x7f55e8108e30 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55ee1ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:03.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55ee1ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51682/0 (socket says 172.21.15.32:51682) 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55ee1ec700 1 -- 172.21.15.32:0/3297259224 learned_addr learned my addr 172.21.15.32:0/3297259224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55ee1ec700 1 -- 172.21.15.32:0/3297259224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e819c730 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55ee1ec700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f55e4000c00 tx=0x7f55e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e4003aa0 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e401b8f0 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e819c9c0 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e4023870 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.038+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e819f6b0 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.039+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f55e4023b70 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.039+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e80640a0 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.039+0000 7f55df7fe700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 0x7f55d403bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.040+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f55e404e290 con 0x7f55e8108e30 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.040+0000 7f55ed9eb700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 0x7f55d403bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:03.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.040+0000 7f55ed9eb700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 0x7f55d403bc20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f55d8006fd0 tx=0x7f55d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.045+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55e401f030 con 0x7f55e8108e30 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.252+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f55e819fd10 con 0x7f55e8108e30 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.253+0000 7f55df7fe700 1 -- 172.21.15.32:0/3297259224 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f55e4018be0 con 0x7f55e8108e30 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 msgr2=0x7f55d403bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 0x7f55d403bc20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f55d8006fd0 tx=0x7f55d8006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 msgr2=0x7f55e819c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f55e4000c00 tx=0x7f55e4011aa0 comp rx=0 tx=0).stop 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 shutdown_connections 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f55d4039760 0x7f55d403bc20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 --2- 172.21.15.32:0/3297259224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8108e30 0x7f55e819c1f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 >> 172.21.15.32:0/3297259224 conn(0x7f55e8102860 msgr2=0x7f55e8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 shutdown_connections 2024-04-23T18:06:03.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:03.255+0000 7f55f0450700 1 -- 172.21.15.32:0/3297259224 wait complete. 2024-04-23T18:06:05.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a7a273700 1 Processor -- start 2024-04-23T18:06:05.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a7a273700 1 -- start start 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a7a273700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a7a273700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74107c40 con 0x7f4a74107250 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51686/0 (socket says 172.21.15.32:51686) 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.217+0000 7f4a73fff700 1 -- 172.21.15.32:0/1823087019 learned_addr learned my addr 172.21.15.32:0/1823087019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a73fff700 1 -- 172.21.15.32:0/1823087019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a74108450 con 0x7f4a74107250 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a73fff700 1 --2- 172.21.15.32:0/1823087019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0098a0 tx=0x7f4a5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef91828135db8429 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a72ffd700 1 -- 172.21.15.32:0/1823087019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a5c004160 con 0x7f4a74107250 2024-04-23T18:06:05.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a72ffd700 1 -- 172.21.15.32:0/1823087019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a5c004300 con 0x7f4a74107250 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a72ffd700 1 -- 172.21.15.32:0/1823087019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a5c011a20 con 0x7f4a74107250 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a7a273700 1 -- 172.21.15.32:0/1823087019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 msgr2=0x7f4a74107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a7a273700 1 --2- 172.21.15.32:0/1823087019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0098a0 tx=0x7f4a5c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a7a273700 1 -- 172.21.15.32:0/1823087019 shutdown_connections 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a7a273700 1 --2- 172.21.15.32:0/1823087019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a74107670 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.218+0000 7f4a7a273700 1 -- 172.21.15.32:0/1823087019 >> 172.21.15.32:0/1823087019 conn(0x7f4a741027f0 msgr2=0x7f4a74104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.219+0000 7f4a7a273700 1 -- 172.21.15.32:0/1823087019 shutdown_connections 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.219+0000 7f4a7a273700 1 -- 172.21.15.32:0/1823087019 wait complete. 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.219+0000 7f4a7a273700 1 Processor -- start 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.219+0000 7f4a7a273700 1 -- start start 2024-04-23T18:06:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a7a273700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a7a273700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7410b800 con 0x7f4a74107250 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51700/0 (socket says 172.21.15.32:51700) 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a73fff700 1 -- 172.21.15.32:0/463589195 learned_addr learned my addr 172.21.15.32:0/463589195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a73fff700 1 -- 172.21.15.32:0/463589195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7419c710 con 0x7f4a74107250 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a73fff700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0116e0 tx=0x7f4a5c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a5c0039e0 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7419c9a0 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a5c01b8f0 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a5c023760 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.220+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7419f690 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.221+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a740640a0 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.221+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4a5c01b440 con 0x7f4a74107250 2024-04-23T18:06:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.222+0000 7f4a717fa700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 0x7f4a6003bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:05.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.222+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a5c04e100 con 0x7f4a74107250 2024-04-23T18:06:05.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.222+0000 7f4a737fe700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 0x7f4a6003bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:05.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.222+0000 7f4a737fe700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 0x7f4a6003bbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:05.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.226+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a5c01f030 con 0x7f4a74107250 2024-04-23T18:06:05.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.547+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f4a7402e210 con 0x7f4a74107250 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.550+0000 7f4a717fa700 1 -- 172.21.15.32:0/463589195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f4a5c02c3f0 con 0x7f4a74107250 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 msgr2=0x7f4a6003bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 0x7f4a6003bbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 msgr2=0x7f4a7419c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0116e0 tx=0x7f4a5c011aa0 comp rx=0 tx=0).stop 2024-04-23T18:06:05.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 shutdown_connections 2024-04-23T18:06:05.106 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4a60039710 0x7f4a6003bbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:05.106 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 --2- 172.21.15.32:0/463589195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a74107250 0x7f4a7419c1d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:05.107 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 >> 172.21.15.32:0/463589195 conn(0x7f4a741027f0 msgr2=0x7f4a741035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:05.107 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 shutdown_connections 2024-04-23T18:06:05.107 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-23T18:06:04.552+0000 7f4a7a273700 1 -- 172.21.15.32:0/463589195 wait complete. 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-04-23T18:06:05.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.109 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T18:06:05.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:05.109 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-04-23T18:06:05.140 INFO:tasks.cephadm:Fetching config... 2024-04-23T18:06:05.140 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:06:05.140 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T18:06:05.156 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T18:06:05.157 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:06:05.157 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T18:06:05.212 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T18:06:05.213 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:06:05.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T18:06:05.277 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T18:06:05.277 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:06:05.277 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T18:06:05.333 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T18:06:05.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:06:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:05.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:06:05.462 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:05.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:06:05.516 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNPXxUY61cQFc/HJs2O4fm8a/DHVRDrDiYd1h+oOHQmTAjxa10o3Jx5fzZy1N9WUtzACS0GLuj5sHH8xsxUN+nflAysNB9/s4Dv7HlP63tH6EzPf5cwlzA5goSztaeaHeq42hLILxZGLLzosvQYrtSt1KKxQx5G8CUoHnA3bQvuEDMhIAzwOR1b+e00ZTuRYmQYlJwUe5DoxY5o3TaeVEjqpiswIHCL03w/nGIZtfW6l34jWm13xHb+daZKabq/3h0C6Yy8AcSzwhW7TSTlfjRt9NmwrXLY70nipz9VmZlWT7PiR+8mscvee4593Vn4K198LQqR26bkbqG7UCyn5cgLmoEbpe0EbxeyEyg2O+j5a/LF2mYCQ9uTB4AxFpMBZRlyXxFkqEjK2qbB5LaWDEhciWDNX+Q11oj7PSrJ8qFfq5H70YE/HjQzfY9nwLtMjqhxn/MJX1x5G6wMpwSsAr8UIR5pvS+2qaNxl/dyszKlBaRq2KpOV6kATP+++YPLbk= ceph-cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:05.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T18:06:05.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/463589195' entity='client.admin' 2024-04-23T18:06:05.773 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:08.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 -- 172.21.15.32:0/1835669561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 msgr2=0x7efc601053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 --2- 172.21.15.32:0/1835669561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc601053c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009910 tx=0x7efc5c009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 -- 172.21.15.32:0/1835669561 shutdown_connections 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 --2- 172.21.15.32:0/1835669561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc601053c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 -- 172.21.15.32:0/1835669561 >> 172.21.15.32:0/1835669561 conn(0x7efc600fc940 msgr2=0x7efc600fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 -- 172.21.15.32:0/1835669561 shutdown_connections 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 -- 172.21.15.32:0/1835669561 wait complete. 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.267+0000 7efc686fe700 1 Processor -- start 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc686fe700 1 -- start start 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc686fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc686fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc601083e0 con 0x7efc60102fd0 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc6649a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:08.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc6649a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51532/0 (socket says 172.21.15.32:51532) 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc6649a700 1 -- 172.21.15.32:0/3375435329 learned_addr learned my addr 172.21.15.32:0/3375435329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc6649a700 1 -- 172.21.15.32:0/3375435329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc60197fa0 con 0x7efc60102fd0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc6649a700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc5c005b40 tx=0x7efc5c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c027070 con 0x7efc60102fd0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.268+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc5c00be10 con 0x7efc60102fd0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.269+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c022de0 con 0x7efc60102fd0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.269+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc60198230 con 0x7efc60102fd0 2024-04-23T18:06:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.269+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6019af20 con 0x7efc60102fd0 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.269+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efc5c02a070 con 0x7efc60102fd0 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.270+0000 7efc577fe700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 0x7efc4c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.270+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc60108b60 con 0x7efc60102fd0 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.270+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efc5c01e030 con 0x7efc60102fd0 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.270+0000 7efc65c99700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 0x7efc4c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.270+0000 7efc65c99700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 0x7efc4c03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efc50006fd0 tx=0x7efc50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:08.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.275+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc5c013070 con 0x7efc60102fd0 2024-04-23T18:06:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.484+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7efc6002e090 con 0x7efc60102fd0 2024-04-23T18:06:08.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.486+0000 7efc577fe700 1 -- 172.21.15.32:0/3375435329 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7efc5c0292c0 con 0x7efc60102fd0 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 msgr2=0x7efc4c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 0x7efc4c03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efc50006fd0 tx=0x7efc50006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 msgr2=0x7efc60197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc5c005b40 tx=0x7efc5c004f80 comp rx=0 tx=0).stop 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 shutdown_connections 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.489+0000 7efc686fe700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efc4c039650 0x7efc4c03bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.490+0000 7efc686fe700 1 --2- 172.21.15.32:0/3375435329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60102fd0 0x7efc60197a60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.490+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 >> 172.21.15.32:0/3375435329 conn(0x7efc600fc940 msgr2=0x7efc600fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.490+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 shutdown_connections 2024-04-23T18:06:08.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:08.490+0000 7efc686fe700 1 -- 172.21.15.32:0/3375435329 wait complete. 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3375435329' entity='client.admin' 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:09.578 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:09.579 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:09.605 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T18:06:09.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T18:06:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 -- 172.21.15.32:0/445063983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 msgr2=0x7f0d20102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 --2- 172.21.15.32:0/445063983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d20102900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c009910 tx=0x7f0d1c009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 -- 172.21.15.32:0/445063983 shutdown_connections 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 --2- 172.21.15.32:0/445063983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d20102900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 -- 172.21.15.32:0/445063983 >> 172.21.15.32:0/445063983 conn(0x7f0d200fda40 msgr2=0x7f0d200ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 -- 172.21.15.32:0/445063983 shutdown_connections 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.359+0000 7f0d285f5700 1 -- 172.21.15.32:0/445063983 wait complete. 2024-04-23T18:06:11.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d285f5700 1 Processor -- start 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d285f5700 1 -- start start 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d285f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d285f5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20108420 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d26391700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d26391700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51546/0 (socket says 172.21.15.32:51546) 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d26391700 1 -- 172.21.15.32:0/2494945246 learned_addr learned my addr 172.21.15.32:0/2494945246 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d26391700 1 -- 172.21.15.32:0/2494945246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d20076bf0 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.360+0000 7f0d26391700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c005800 tx=0x7f0d1c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.361+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d1c027070 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.361+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d1c00be10 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.361+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d20074e30 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.361+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d1c00f460 con 0x7f0d201024e0 2024-04-23T18:06:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.361+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d20075310 con 0x7f0d201024e0 2024-04-23T18:06:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0d1c02a070 con 0x7f0d201024e0 2024-04-23T18:06:11.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d20108ba0 con 0x7f0d201024e0 2024-04-23T18:06:11.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d177fe700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 0x7f0d0c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:11.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0d1c01e030 con 0x7f0d201024e0 2024-04-23T18:06:11.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d25b90700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 0x7f0d0c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:11.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.362+0000 7f0d25b90700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 0x7f0d0c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0d10006fd0 tx=0x7f0d10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:11.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.367+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d1c013070 con 0x7f0d201024e0 2024-04-23T18:06:11.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.582+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f0d20190e80 con 0x7f0d0c0396b0 2024-04-23T18:06:11.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.586+0000 7f0d177fe700 1 -- 172.21.15.32:0/2494945246 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d20190e80 con 0x7f0d0c0396b0 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 msgr2=0x7f0d0c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 0x7f0d0c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0d10006fd0 tx=0x7f0d10006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 msgr2=0x7f0d200766b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c005800 tx=0x7f0d1c009f60 comp rx=0 tx=0).stop 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 shutdown_connections 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0d0c0396b0 0x7f0d0c03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 --2- 172.21.15.32:0/2494945246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d201024e0 0x7f0d200766b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 >> 172.21.15.32:0/2494945246 conn(0x7f0d200fda40 msgr2=0x7f0d200fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 shutdown_connections 2024-04-23T18:06:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:11.588+0000 7f0d285f5700 1 -- 172.21.15.32:0/2494945246 wait complete. 2024-04-23T18:06:12.356 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi040 2024-04-23T18:06:12.356 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:06:12.356 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:06:12.376 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:06:12.376 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:12.434 INFO:tasks.cephadm:Adding host smithi040 to orchestrator... 2024-04-23T18:06:12.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch host add smithi040 2024-04-23T18:06:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:12.869 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:12.869 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:12.869 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:12.869 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:12.869 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:12.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-23T18:06:12.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:12.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:12.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:12.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:13 smithi032 ceph-mon[29221]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:13 smithi032 ceph-mon[29221]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:06:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 -- 172.21.15.32:0/1475743826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 msgr2=0x7f7450103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 --2- 172.21.15.32:0/1475743826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450103a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7444009910 tx=0x7f7444009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 -- 172.21.15.32:0/1475743826 shutdown_connections 2024-04-23T18:06:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 --2- 172.21.15.32:0/1475743826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450103a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 -- 172.21.15.32:0/1475743826 >> 172.21.15.32:0/1475743826 conn(0x7f74500fcea0 msgr2=0x7f74500ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 -- 172.21.15.32:0/1475743826 shutdown_connections 2024-04-23T18:06:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.113+0000 7f7457126700 1 -- 172.21.15.32:0/1475743826 wait complete. 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7457126700 1 Processor -- start 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7457126700 1 -- start start 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7457126700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7457126700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f745010c9e0 con 0x7f74501036b0 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7454ec2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7454ec2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51556/0 (socket says 172.21.15.32:51556) 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7454ec2700 1 -- 172.21.15.32:0/3411125561 learned_addr learned my addr 172.21.15.32:0/3411125561 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.114+0000 7f7454ec2700 1 -- 172.21.15.32:0/3411125561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7450198490 con 0x7f74501036b0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f7454ec2700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74440056d0 tx=0x7f74440057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7444026070 con 0x7f74501036b0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7450198720 con 0x7f74501036b0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7444004370 con 0x7f74501036b0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74440039b0 con 0x7f74501036b0 2024-04-23T18:06:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.115+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7450198b80 con 0x7f74501036b0 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.116+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7444029070 con 0x7f74501036b0 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.116+0000 7f744dffb700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 0x7f744003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.116+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f744401e070 con 0x7f74501036b0 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.116+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745006b130 con 0x7f74501036b0 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.116+0000 7f744ffff700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 0x7f744003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.117+0000 7f744ffff700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 0x7f744003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f743c006fd0 tx=0x7f743c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.121+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7444030020 con 0x7f74501036b0 2024-04-23T18:06:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:14.337+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}) -- 0x7f74501992b0 con 0x7f7440039650 2024-04-23T18:06:15.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:14 smithi032 ceph-mon[29221]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:16.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:15 smithi032 ceph-mon[29221]: Deploying cephadm binary to smithi040 2024-04-23T18:06:17.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:16 smithi032 ceph-mon[29221]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:18.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.569+0000 7f744dffb700 1 -- 172.21.15.32:0/3411125561 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f74501992b0 con 0x7f7440039650 2024-04-23T18:06:18.565 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi040' with addr '172.21.15.40' 2024-04-23T18:06:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 msgr2=0x7f744003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 0x7f744003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f743c006fd0 tx=0x7f743c006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 msgr2=0x7f7450197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74440056d0 tx=0x7f74440057b0 comp rx=0 tx=0).stop 2024-04-23T18:06:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 shutdown_connections 2024-04-23T18:06:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7440039650 0x7f744003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 --2- 172.21.15.32:0/3411125561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74501036b0 0x7f7450197f50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 >> 172.21.15.32:0/3411125561 conn(0x7f74500fcea0 msgr2=0x7f7450104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 shutdown_connections 2024-04-23T18:06:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:18.572+0000 7f7457126700 1 -- 172.21.15.32:0/3411125561 wait complete. 2024-04-23T18:06:19.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:06:19.417 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:19.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:19 smithi032 ceph-mon[29221]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:19.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:19 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:19.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:19 smithi032 ceph-mon[29221]: Added host smithi040 2024-04-23T18:06:19.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:19 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:19.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:19 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:20.714 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 -- 172.21.15.32:0/1652245466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 msgr2=0x7f50801088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 --2- 172.21.15.32:0/1652245466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f50801088c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5070009910 tx=0x7f5070009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 -- 172.21.15.32:0/1652245466 shutdown_connections 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 --2- 172.21.15.32:0/1652245466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f50801088c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 -- 172.21.15.32:0/1652245466 >> 172.21.15.32:0/1652245466 conn(0x7f5080076ea0 msgr2=0x7f50800772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 -- 172.21.15.32:0/1652245466 shutdown_connections 2024-04-23T18:06:20.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.826+0000 7f5084e3c700 1 -- 172.21.15.32:0/1652245466 wait complete. 2024-04-23T18:06:20.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f5084e3c700 1 Processor -- start 2024-04-23T18:06:20.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f5084e3c700 1 -- start start 2024-04-23T18:06:20.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f5084e3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:20.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f5084e3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f508010c920 con 0x7f50801084e0 2024-04-23T18:06:20.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f507e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f507e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37970/0 (socket says 172.21.15.32:37970) 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f507e59c700 1 -- 172.21.15.32:0/2991392252 learned_addr learned my addr 172.21.15.32:0/2991392252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f507e59c700 1 -- 172.21.15.32:0/2991392252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5080103350 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.827+0000 7f507e59c700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5070006010 tx=0x7f5070005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5070026070 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507000bc00 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5080101a00 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5070003910 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5080101ec0 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.828+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5070029070 con 0x7f50801084e0 2024-04-23T18:06:20.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.829+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5080103b90 con 0x7f50801084e0 2024-04-23T18:06:20.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.829+0000 7f50777fe700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 0x7f506c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:20.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.829+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f507001e070 con 0x7f50801084e0 2024-04-23T18:06:20.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.829+0000 7f5077fff700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 0x7f506c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:20.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.829+0000 7f5077fff700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 0x7f506c03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5068006fd0 tx=0x7f5068006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:20.834+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5070013070 con 0x7f50801084e0 2024-04-23T18:06:21.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.049+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5080102a00 con 0x7f506c039650 2024-04-23T18:06:21.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.050+0000 7f50777fe700 1 -- 172.21.15.32:0/2991392252 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f5080102a00 con 0x7f506c039650 2024-04-23T18:06:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:21.045 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}] 2024-04-23T18:06:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 msgr2=0x7f506c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 0x7f506c03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5068006fd0 tx=0x7f5068006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 msgr2=0x7f5080101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5070006010 tx=0x7f5070005780 comp rx=0 tx=0).stop 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 shutdown_connections 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f506c039650 0x7f506c03bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 --2- 172.21.15.32:0/2991392252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50801084e0 0x7f5080101460 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.052+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 >> 172.21.15.32:0/2991392252 conn(0x7f5080076ea0 msgr2=0x7f50800fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.053+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 shutdown_connections 2024-04-23T18:06:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:21.053+0000 7f5084e3c700 1 -- 172.21.15.32:0/2991392252 wait complete. 2024-04-23T18:06:21.631 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi064 2024-04-23T18:06:21.631 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:06:21.631 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:06:21.651 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:06:21.651 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:21.711 INFO:tasks.cephadm:Adding host smithi064 to orchestrator... 2024-04-23T18:06:21.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch host add smithi064 2024-04-23T18:06:21.955 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:22.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:21 smithi032 ceph-mon[29221]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:22.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:21 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:22.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:22 smithi032 ceph-mon[29221]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:06:23.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- 172.21.15.32:0/2011424359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 msgr2=0x7f77e41019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/2011424359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e41019d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f77d4009910 tx=0x7f77d4009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- 172.21.15.32:0/2011424359 shutdown_connections 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/2011424359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e41019d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- 172.21.15.32:0/2011424359 >> 172.21.15.32:0/2011424359 conn(0x7f77e40fcea0 msgr2=0x7f77e40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- 172.21.15.32:0/2011424359 shutdown_connections 2024-04-23T18:06:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- 172.21.15.32:0/2011424359 wait complete. 2024-04-23T18:06:23.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 Processor -- start 2024-04-23T18:06:23.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.587+0000 7f77ea3e1700 1 -- start start 2024-04-23T18:06:23.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77ea3e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77ea3e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77e410b870 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37994/0 (socket says 172.21.15.32:37994) 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e3fff700 1 -- 172.21.15.32:0/686123717 learned_addr learned my addr 172.21.15.32:0/686123717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e3fff700 1 -- 172.21.15.32:0/686123717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e4104f80 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e3fff700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f77d40056d0 tx=0x7f77d40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d4026070 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e4103630 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d400bc00 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.588+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d4003920 con 0x7f77e41015f0 2024-04-23T18:06:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.589+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e4103af0 con 0x7f77e41015f0 2024-04-23T18:06:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f77d4029070 con 0x7f77e41015f0 2024-04-23T18:06:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e4102ad0 con 0x7f77e41015f0 2024-04-23T18:06:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77e17fa700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 0x7f77cc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f77d401e070 con 0x7f77e41015f0 2024-04-23T18:06:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77e37fe700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 0x7f77cc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.590+0000 7f77e37fe700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 0x7f77cc03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f77d8006fd0 tx=0x7f77d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.595+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77d4013070 con 0x7f77e41015f0 2024-04-23T18:06:23.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:23.809+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}) -- 0x7f77e41045c0 con 0x7f77cc039650 2024-04-23T18:06:23.805 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:23 smithi032 ceph-mon[29221]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:25.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:25.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:25.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:25 smithi032 ceph-mon[29221]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:25 smithi032 ceph-mon[29221]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-23T18:06:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:25 smithi032 ceph-mon[29221]: Deploying cephadm binary to smithi064 2024-04-23T18:06:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:25 smithi032 ceph-mon[29221]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:25 smithi032 ceph-mon[29221]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:27 smithi032 ceph-mon[29221]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.074+0000 7f77e17fa700 1 -- 172.21.15.32:0/686123717 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f77e41045c0 con 0x7f77cc039650 2024-04-23T18:06:28.069 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi064' with addr '172.21.15.64' 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 msgr2=0x7f77cc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 0x7f77cc03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f77d8006fd0 tx=0x7f77d8006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 msgr2=0x7f77e4103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f77d40056d0 tx=0x7f77d40057b0 comp rx=0 tx=0).stop 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 shutdown_connections 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc039650 0x7f77cc03bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 --2- 172.21.15.32:0/686123717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e41015f0 0x7f77e4103090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 >> 172.21.15.32:0/686123717 conn(0x7f77e40fcea0 msgr2=0x7f77e406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 shutdown_connections 2024-04-23T18:06:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:28.076+0000 7f77ea3e1700 1 -- 172.21.15.32:0/686123717 wait complete. 2024-04-23T18:06:28.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:06:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:28.946 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:29.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:29 smithi032 ceph-mon[29221]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:29.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:29 smithi032 ceph-mon[29221]: Added host smithi064 2024-04-23T18:06:29.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 -- 172.21.15.32:0/4017733542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 msgr2=0x7f92c41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/4017733542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c41088c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009910 tx=0x7f92c0009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 -- 172.21.15.32:0/4017733542 shutdown_connections 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/4017733542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c41088c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 -- 172.21.15.32:0/4017733542 >> 172.21.15.32:0/4017733542 conn(0x7f92c4076ea0 msgr2=0x7f92c40772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 -- 172.21.15.32:0/4017733542 shutdown_connections 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.157+0000 7f92cc2dd700 1 -- 172.21.15.32:0/4017733542 wait complete. 2024-04-23T18:06:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92cc2dd700 1 Processor -- start 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92cc2dd700 1 -- start start 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92cc2dd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92cc2dd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410c920 con 0x7f92c41084e0 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92ca079700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92ca079700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49818/0 (socket says 172.21.15.32:49818) 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92ca079700 1 -- 172.21.15.32:0/1971878655 learned_addr learned my addr 172.21.15.32:0/1971878655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.158+0000 7f92ca079700 1 -- 172.21.15.32:0/1971878655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c41983f0 con 0x7f92c41084e0 2024-04-23T18:06:30.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92ca079700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92c00056d0 tx=0x7f92c00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c0026070 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c41986e0 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c000bba0 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c0003920 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.159+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c419c780 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f92c0029070 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c4103b90 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92bb7fe700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 0x7f92b003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f92c001e070 con 0x7f92c41084e0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92c9878700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 0x7f92b003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.160+0000 7f92c9878700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 0x7f92b003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f92b4006fd0 tx=0x7f92b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.165+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92c0013070 con 0x7f92c41084e0 2024-04-23T18:06:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.381+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92c4199210 con 0x7f92b0039650 2024-04-23T18:06:30.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.382+0000 7f92bb7fe700 1 -- 172.21.15.32:0/1971878655 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f92c4199210 con 0x7f92b0039650 2024-04-23T18:06:30.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:06:30.377 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}] 2024-04-23T18:06:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 msgr2=0x7f92b003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 0x7f92b003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f92b4006fd0 tx=0x7f92b4006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 msgr2=0x7f92c4197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92c00056d0 tx=0x7f92c00057b0 comp rx=0 tx=0).stop 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 shutdown_connections 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f92b0039650 0x7f92b003bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 --2- 172.21.15.32:0/1971878655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c41084e0 0x7f92c4197eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 >> 172.21.15.32:0/1971878655 conn(0x7f92c4076ea0 msgr2=0x7f92c40fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 shutdown_connections 2024-04-23T18:06:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:30.384+0000 7f92cc2dd700 1 -- 172.21.15.32:0/1971878655 wait complete. 2024-04-23T18:06:30.929 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T18:06:30.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T18:06:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:31.938 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:31 smithi032 ceph-mon[29221]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:31.938 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:31 smithi032 ceph-mon[29221]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:06:31.938 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 -- 172.21.15.32:0/567111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 msgr2=0x7f29cc108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 --2- 172.21.15.32:0/567111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc108930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009910 tx=0x7f29bc009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 -- 172.21.15.32:0/567111719 shutdown_connections 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 --2- 172.21.15.32:0/567111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc108930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 -- 172.21.15.32:0/567111719 >> 172.21.15.32:0/567111719 conn(0x7f29cc0fdf70 msgr2=0x7f29cc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 -- 172.21.15.32:0/567111719 shutdown_connections 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.705+0000 7f29d2a11700 1 -- 172.21.15.32:0/567111719 wait complete. 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29d2a11700 1 Processor -- start 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29d2a11700 1 -- start start 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29d2a11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29d2a11700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc10c9c0 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49840/0 (socket says 172.21.15.32:49840) 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.706+0000 7f29cbfff700 1 -- 172.21.15.32:0/64335723 learned_addr learned my addr 172.21.15.32:0/64335723 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29cbfff700 1 -- 172.21.15.32:0/64335723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc076c30 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29cbfff700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29bc006010 tx=0x7f29bc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29bc026070 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29bc00bc00 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc075280 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29bc003910 con 0x7f29cc108550 2024-04-23T18:06:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.707+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc075740 con 0x7f29cc108550 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.708+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f29bc029070 con 0x7f29cc108550 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.708+0000 7f29c97fa700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 0x7f29b403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.708+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f29bc01e070 con 0x7f29cc108550 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.708+0000 7f29cb7fe700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 0x7f29b403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.708+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc103bc0 con 0x7f29cc108550 2024-04-23T18:06:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.709+0000 7f29cb7fe700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 0x7f29b403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f29c0006fd0 tx=0x7f29c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:32.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.713+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29bc030070 con 0x7f29cc108550 2024-04-23T18:06:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:32.929+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f29cc076270 con 0x7f29cc108550 2024-04-23T18:06:33.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.786+0000 7f29c97fa700 1 -- 172.21.15.32:0/64335723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f29bc013070 con 0x7f29cc108550 2024-04-23T18:06:33.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.788+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 msgr2=0x7f29b403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.788+0000 7f29d2a11700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 0x7f29b403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f29c0006fd0 tx=0x7f29c0006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.788+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 msgr2=0x7f29cc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.788+0000 7f29d2a11700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29bc006010 tx=0x7f29bc005780 comp rx=0 tx=0).stop 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 shutdown_connections 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29b40396b0 0x7f29b403bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 --2- 172.21.15.32:0/64335723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc108550 0x7f29cc074d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 >> 172.21.15.32:0/64335723 conn(0x7f29cc0fdf70 msgr2=0x7f29cc0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 shutdown_connections 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:06:33.789+0000 7f29d2a11700 1 -- 172.21.15.32:0/64335723 wait complete. 2024-04-23T18:06:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-04-23T18:06:34.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:33 smithi032 ceph-mon[29221]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:34.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:33 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/64335723' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:06:34.477 INFO:tasks.cephadm:Adding mon.a on smithi032 2024-04-23T18:06:34.477 INFO:tasks.cephadm:Adding mon.b on smithi040 2024-04-23T18:06:34.477 INFO:tasks.cephadm:Adding mon.c on smithi064 2024-04-23T18:06:34.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c' 2024-04-23T18:06:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:35.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/64335723' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:06:35.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:36.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.211+0000 7f19a0658700 1 -- 172.21.15.64:0/523221473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 msgr2=0x7f19981088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.211+0000 7f19a0658700 1 --2- 172.21.15.64:0/523221473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f19981088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1994009910 tx=0x7f1994009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- 172.21.15.64:0/523221473 shutdown_connections 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 --2- 172.21.15.64:0/523221473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f19981088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- 172.21.15.64:0/523221473 >> 172.21.15.64:0/523221473 conn(0x7f19980fdf80 msgr2=0x7f19981003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- 172.21.15.64:0/523221473 shutdown_connections 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- 172.21.15.64:0/523221473 wait complete. 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 Processor -- start 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- start start 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.212+0000 7f19a0658700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f199810c940 con 0x7f19981084e0 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f199e3f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:36.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f199e3f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.64:52768/0 (socket says 172.21.15.64:52768) 2024-04-23T18:06:36.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f199e3f4700 1 -- 172.21.15.64:0/2788236683 learned_addr learned my addr 172.21.15.64:0/2788236683 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:06:36.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f199e3f4700 1 -- 172.21.15.64:0/2788236683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19981983f0 con 0x7f19981084e0 2024-04-23T18:06:36.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f199e3f4700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f19940056d0 tx=0x7f19940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:36.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.213+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1994026070 con 0x7f19981084e0 2024-04-23T18:06:36.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f199400bba0 con 0x7f19981084e0 2024-04-23T18:06:36.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19981986e0 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1994003920 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199819c780 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1994029070 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.214+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1998103b90 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.215+0000 7f198b7fe700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 0x7f198403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.215+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f199401e070 con 0x7f19981084e0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.215+0000 7f199dbf3700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 0x7f198403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:36.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.215+0000 7f199dbf3700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 0x7f198403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f198c006fd0 tx=0x7f198c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:36.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.220+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1994013070 con 0x7f19981084e0 2024-04-23T18:06:36.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:36.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:36.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.442+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c", "target": ["mon-mgr", ""]}) -- 0x7f1998199210 con 0x7f1984039650 2024-04-23T18:06:36.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.449+0000 7f198b7fe700 1 -- 172.21.15.64:0/2788236683 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1998199210 con 0x7f1984039650 2024-04-23T18:06:36.444 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled mon update... 2024-04-23T18:06:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 msgr2=0x7f198403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 0x7f198403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f198c006fd0 tx=0x7f198c006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 msgr2=0x7f1998197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f19940056d0 tx=0x7f19940057b0 comp rx=0 tx=0).stop 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 shutdown_connections 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f1984039650 0x7f198403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 --2- 172.21.15.64:0/2788236683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19981084e0 0x7f1998197eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 >> 172.21.15.64:0/2788236683 conn(0x7f19980fdf80 msgr2=0x7f199810a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 shutdown_connections 2024-04-23T18:06:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:36.451+0000 7f19a0658700 1 -- 172.21.15.64:0/2788236683 wait complete. 2024-04-23T18:06:37.051 DEBUG:teuthology.orchestra.run.smithi040:mon.b> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T18:06:37.054 DEBUG:teuthology.orchestra.run.smithi064:mon.c> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T18:06:37.056 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:06:37.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:06:37.748 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:37.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:37.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:37.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c;count:3 2024-04-23T18:06:37.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:37.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:37.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:37 smithi032 ceph-mon[29221]: Deploying daemon mon.c on smithi064 2024-04-23T18:06:39.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:39 smithi032 ceph-mon[29221]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:40.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 -- 172.21.15.64:0/1493612191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 msgr2=0x7f77ac1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 --2- 172.21.15.64:0/1493612191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1019d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77a8009910 tx=0x7f77a8009c20 comp rx=0 tx=0).stop 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 -- 172.21.15.64:0/1493612191 shutdown_connections 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 --2- 172.21.15.64:0/1493612191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1019d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 -- 172.21.15.64:0/1493612191 >> 172.21.15.64:0/1493612191 conn(0x7f77ac076ea0 msgr2=0x7f77ac0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 -- 172.21.15.64:0/1493612191 shutdown_connections 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.534+0000 7f77b3d92700 1 -- 172.21.15.64:0/1493612191 wait complete. 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b3d92700 1 Processor -- start 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b3d92700 1 -- start start 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b3d92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b3d92700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac10b820 con 0x7f77ac1015f0 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b1b2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:40.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.535+0000 7f77b1b2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.64:50938/0 (socket says 172.21.15.64:50938) 2024-04-23T18:06:40.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77b1b2e700 1 -- 172.21.15.64:0/2587252905 learned_addr learned my addr 172.21.15.64:0/2587252905 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:06:40.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77b1b2e700 1 -- 172.21.15.64:0/2587252905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac104f90 con 0x7f77ac1015f0 2024-04-23T18:06:40.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77b1b2e700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f77a80056d0 tx=0x7f77a80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77a8026070 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ac103640 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77a800bc00 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ac103b00 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.536+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77a8003920 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.538+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77ac102a80 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.538+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f77a8029070 con 0x7f77ac1015f0 2024-04-23T18:06:40.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.538+0000 7f77a2ffd700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 0x7f779803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:40.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.538+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f77a801e070 con 0x7f77ac1015f0 2024-04-23T18:06:40.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.538+0000 7f77b132d700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 0x7f779803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:40.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.539+0000 7f77b132d700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 0x7f779803bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:40.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.544+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77a8013070 con 0x7f77ac1015f0 2024-04-23T18:06:40.850 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:40 smithi064 systemd[1]: Starting Ceph mon.c for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:06:40.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.855+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f77ac1045d0 con 0x7f77ac1015f0 2024-04-23T18:06:40.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.856+0000 7f77a2ffd700 1 -- 172.21.15.64:0/2587252905 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f77a8030090 con 0x7f77ac1015f0 2024-04-23T18:06:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:06:40.852 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","modified":"2024-04-23T18:04:48.613148Z","created":"2024-04-23T18:04:48.613148Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T18:06:40.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.858+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 msgr2=0x7f779803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:40.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.858+0000 7f77b3d92700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 0x7f779803bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c006e40 comp rx=0 tx=0).stop 2024-04-23T18:06:40.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.858+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 msgr2=0x7f77ac1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.858+0000 7f77b3d92700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f77a80056d0 tx=0x7f77a80057b0 comp rx=0 tx=0).stop 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 shutdown_connections 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77980396b0 0x7f779803bb70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 --2- 172.21.15.64:0/2587252905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77ac1015f0 0x7f77ac1030a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 >> 172.21.15.64:0/2587252905 conn(0x7f77ac076ea0 msgr2=0x7f77ac069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 shutdown_connections 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:40.859+0000 7f77b3d92700 1 -- 172.21.15.64:0/2587252905 wait complete. 2024-04-23T18:06:40.854 INFO:teuthology.orchestra.run.smithi064.stderr:dumped monmap epoch 1 2024-04-23T18:06:41.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:40 smithi064 podman[33028]: 2024-04-23 18:06:40.890748581 +0000 UTC m=+0.019706420 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:06:41.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:40 smithi064 podman[33028]: 2024-04-23 18:06:40.988212081 +0000 UTC m=+0.117169903 container create 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:06:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:41 smithi032 ceph-mon[29221]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:41 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/2587252905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:06:42.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 podman[33028]: 2024-04-23 18:06:41.912984567 +0000 UTC m=+1.041942390 container init 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:06:42.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 podman[33028]: 2024-04-23 18:06:41.91665506 +0000 UTC m=+1.045612883 container start 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T18:06:42.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:06:42.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:06:42.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pidfile_write: ignore empty --pid-file 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: load: jerasure load: lrc 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Git sha 0 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: DB SUMMARY 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: DB Session ID: 285WPCSM7EMNKA9SQ3W1 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:06:42.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.env: 0x557aef8eacc0 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.info_log: 0x557af0bf7be0 2024-04-23T18:06:42.175 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.statistics: (nil) 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.use_fsync: 0 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.db_log_dir: 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.wal_dir: 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.write_buffer_manager: 0x557af0b8b0e0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.unordered_write: 0 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:06:42.179 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.row_cache: None 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.wal_filter: None 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.wal_compression: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:06:42.180 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:06:42.181 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_open_files: -1 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Compression algorithms supported: 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kZSTD supported: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:06:42.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.merge_operator: 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_filter: None 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557af0bf7dc0) 2024-04-23T18:06:42.183 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_type: 0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_index_type: 0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_shortening: 1 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: checksum: 4 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: no_block_cache: 0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache: 0x557af0bbd1f0 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_options: 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: capacity : 536870912 2024-04-23T18:06:42.184 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_shard_bits : 4 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: strict_capacity_limit : 0 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_compressed: (nil) 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: persistent_cache: (nil) 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size: 4096 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size_deviation: 10 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_restart_interval: 16 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_block_restart_interval: 1 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: metadata_block_size: 4096 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: partition_filters: 0 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: use_delta_encoding: 1 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: filter_policy: bloomfilter 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: whole_key_filtering: 1 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: verify_compression: 0 2024-04-23T18:06:42.185 INFO:journalctl@ceph.mon.c.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: format_version: 5 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_index_compression: 1 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_align: 0 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: max_auto_readahead_size: 262144 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: prepopulate_block_cache: 0 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression: NoCompression 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:06:42.186 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.num_levels: 7 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:06:42.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:06:42.188 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:06:42.189 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:06:42.190 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.ttl: 2592000 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:06:42.191 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 832dcf17-7c2c-4137-a8ee-01b2048cf749 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895601949625, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895601950091, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895601, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "832dcf17-7c2c-4137-a8ee-01b2048cf749", "db_session_id": "285WPCSM7EMNKA9SQ3W1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895601950182, "job": 1, "event": "recovery_finished"} 2024-04-23T18:06:42.192 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557af0cd2000 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: DB pointer 0x557af0cbc000 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** DB Stats ** 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.193 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.194 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache BinnedLRUCache@0x557af0bbd1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:06:42.195 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: using public_addr v2:172.21.15.64:0/0 -> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: starting mon.c rank -1 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(???) e0 preinit fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: e1 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: legacy client fscid: -1 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout: No filesystems configured 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T18:06:42.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.197 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Deploying cephadm binary to smithi064 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.198 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Added host smithi064 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/64335723' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/64335723' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.199 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:06:42.200 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c;count:3 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: Deploying daemon mon.c on smithi064 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/2587252905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 ceph-mon[33062]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 bash[33028]: 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f 2024-04-23T18:06:42.201 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:41 smithi064 systemd[1]: Started Ceph mon.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:06:42.650 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:06:42.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:06:42.904 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: mon.a calling monitor election 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: mon.c calling monitor election 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: monmap epoch 2 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: last_changed 2024-04-23T18:06:41.979114+0000 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: min_mon_release 19 (squid) 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: election_strategy: 1 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:47.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: fsmap 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: mgrmap e13: x(active, since 51s) 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:47.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:47.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:06:47.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: mon.a calling monitor election 2024-04-23T18:06:47.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: mon.c calling monitor election 2024-04-23T18:06:47.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:47.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: monmap epoch 2 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: last_changed 2024-04-23T18:06:41.979114+0000 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:47.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: min_mon_release 19 (squid) 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: election_strategy: 1 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: fsmap 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: mgrmap e13: x(active, since 51s) 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:47.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:47.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 -- 172.21.15.64:0/316038474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740035c0 msgr2=0x7fc774005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 --2- 172.21.15.64:0/316038474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740035c0 0x7fc774005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc78c102490 tx=0x7fc77c00feb0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 -- 172.21.15.64:0/316038474 shutdown_connections 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 --2- 172.21.15.64:0/316038474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740035c0 0x7fc774005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 -- 172.21.15.64:0/316038474 >> 172.21.15.64:0/316038474 conn(0x7fc78c0fe050 msgr2=0x7fc78c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 -- 172.21.15.64:0/316038474 shutdown_connections 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.518+0000 7fc791b79700 1 -- 172.21.15.64:0/316038474 wait complete. 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.519+0000 7fc791b79700 1 Processor -- start 2024-04-23T18:06:47.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc791b79700 1 -- start start 2024-04-23T18:06:47.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc791b79700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:47.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc791b79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 0x7fc78c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:47.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc791b79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c10a4b0 con 0x7fc78c1984d0 2024-04-23T18:06:47.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc791b79700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c10a330 con 0x7fc7740035c0 2024-04-23T18:06:47.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:47.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38640/0 (socket says 172.21.15.64:38640) 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 -- 172.21.15.64:0/2966244637 learned_addr learned my addr 172.21.15.64:0/2966244637 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78affd700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 0x7fc78c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 -- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 msgr2=0x7fc78c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 -- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 msgr2=0x7fc78c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:06:47.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.520+0000 7fc78b7fe700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc78affd700 1 -- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 msgr2=0x7fc78c197f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc78affd700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc78affd700 1 -- 172.21.15.64:0/2966244637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78c198b10 con 0x7fc78c1984d0 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc78affd700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 0x7fc78c19c860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc78000b4d0 tx=0x7fc78000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc780011630 con 0x7fc78c1984d0 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc780011c80 con 0x7fc78c1984d0 2024-04-23T18:06:47.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78c19cda0 con 0x7fc78c1984d0 2024-04-23T18:06:47.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc780010da0 con 0x7fc78c1984d0 2024-04-23T18:06:47.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.521+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc78c19d380 con 0x7fc78c1984d0 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.522+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc780018020 con 0x7fc78c1984d0 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.522+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78c19d690 con 0x7fc78c1984d0 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.522+0000 7fc788ff9700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 0x7fc77803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.523+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc78004c200 con 0x7fc78c1984d0 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.523+0000 7fc78b7fe700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 0x7fc77803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.523+0000 7fc78b7fe700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 0x7fc77803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00fa60 tx=0x7fc77c005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:47.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.528+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc78c19d690 con 0x7fc78c1984d0 2024-04-23T18:06:47.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.830+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc78c1992f0 con 0x7fc78c1984d0 2024-04-23T18:06:47.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.830+0000 7fc788ff9700 1 -- 172.21.15.64:0/2966244637 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7fc78001f020 con 0x7fc78c1984d0 2024-04-23T18:06:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:06:47.826 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":2,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","modified":"2024-04-23T18:06:41.979114Z","created":"2024-04-23T18:04:48.613148Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:06:47.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 msgr2=0x7fc77803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:47.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 0x7fc77803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00fa60 tx=0x7fc77c005a90 comp rx=0 tx=0).stop 2024-04-23T18:06:47.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 msgr2=0x7fc78c19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:47.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 0x7fc78c19c860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc78000b4d0 tx=0x7fc78000b890 comp rx=0 tx=0).stop 2024-04-23T18:06:47.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 shutdown_connections 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc778039440 0x7fc77803b900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7740035c0 0x7fc78c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 --2- 172.21.15.64:0/2966244637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc78c1984d0 0x7fc78c19c860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.832+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 >> 172.21.15.64:0/2966244637 conn(0x7fc78c0fe050 msgr2=0x7fc78c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.833+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 shutdown_connections 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:47.833+0000 7fc791b79700 1 -- 172.21.15.64:0/2966244637 wait complete. 2024-04-23T18:06:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:dumped monmap epoch 2 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: Deploying daemon mon.b on smithi040 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/2966244637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:06:48.138 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:48.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:48.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:48.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:48.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: Deploying daemon mon.b on smithi040 2024-04-23T18:06:48.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/2966244637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:06:48.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:49 smithi064 ceph-mon[33062]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:49.445 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:06:49.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:06:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:49 smithi032 ceph-mon[29221]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:06:51.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:51 smithi040 systemd[1]: Started Ceph mon.b for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: mon.a calling monitor election 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: mon.c calling monitor election 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: monmap epoch 3 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: min_mon_release 19 (squid) 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: election_strategy: 1 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:56.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: fsmap 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: mgrmap e13: x(active, since 60s) 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:56.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: mon.a calling monitor election 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: mon.c calling monitor election 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:56.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: monmap epoch 3 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: min_mon_release 19 (squid) 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: election_strategy: 1 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: fsmap 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: mgrmap e13: x(active, since 60s) 2024-04-23T18:06:56.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:06:56.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:56.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2009460722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 msgr2=0x7f2bf40086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2009460722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2bf40086b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2c08103620 tx=0x7f2c0400fe90 comp rx=0 tx=0).stop 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2009460722 shutdown_connections 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2009460722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2bf40086b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2009460722 >> 172.21.15.64:0/2009460722 conn(0x7f2c080fe050 msgr2=0x7f2c08100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2009460722 shutdown_connections 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.156+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2009460722 wait complete. 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 Processor -- start 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 -- start start 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2c08074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:57.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:57.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0810a460 con 0x7f2bf4006240 2024-04-23T18:06:57.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0810a2e0 con 0x7f2c081a43b0 2024-04-23T18:06:57.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0fa84700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0810a5e0 con 0x7f2c08075280 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.157+0000 7f2c0d01f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35922/0 (socket says 172.21.15.64:35922) 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 -- 172.21.15.64:0/2042244629 learned_addr learned my addr 172.21.15.64:0/2042244629 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0e021700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d820700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2c08074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0e021700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 msgr2=0x7f2c081a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0e021700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 msgr2=0x7f2c081a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0e021700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0e021700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:06:57.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 msgr2=0x7f2c081a67a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 msgr2=0x7f2c08074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2c08074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 -- 172.21.15.64:0/2042244629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c081a6ed0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0d01f700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800b780 tx=0x7f2bf800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8004230 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bf80043d0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c081a71c0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8011b20 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.158+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c081a7750 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.159+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2bf8011cc0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.159+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c080640a0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.160+0000 7f2bfeffd700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 0x7f2bec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.160+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2bf804c4b0 con 0x7f2c08075280 2024-04-23T18:06:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.160+0000 7f2c0d820700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 0x7f2bec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:06:57.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.160+0000 7f2c0d820700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 0x7f2bec03bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400fa40 tx=0x7f2c04005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:06:57.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.165+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bf8020500 con 0x7f2c08075280 2024-04-23T18:06:57.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:57 smithi064 ceph-mon[33062]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:57.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:57.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:57.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:57.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:57.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.463+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2c081a7a10 con 0x7f2c08075280 2024-04-23T18:06:57.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.465+0000 7f2bfeffd700 1 -- 172.21.15.64:0/2042244629 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7f2bf80202a0 con 0x7f2c08075280 2024-04-23T18:06:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-23T18:06:57.459 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":3,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","modified":"2024-04-23T18:06:51.031323Z","created":"2024-04-23T18:04:48.613148Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:06:57.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 msgr2=0x7f2bec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:57.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 0x7f2bec03bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400fa40 tx=0x7f2c04005ad0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 msgr2=0x7f2c08075700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800b780 tx=0x7f2bf800ba90 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 shutdown_connections 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2bec039ae0 0x7f2bec03bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c081a43b0 0x7f2c081a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bf4006240 0x7f2c08074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 --2- 172.21.15.64:0/2042244629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c08075280 0x7f2c08075700 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 >> 172.21.15.64:0/2042244629 conn(0x7f2c080fe050 msgr2=0x7f2c08104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 shutdown_connections 2024-04-23T18:06:57.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:06:57.467+0000 7f2c0fa84700 1 -- 172.21.15.64:0/2042244629 wait complete. 2024-04-23T18:06:57.463 INFO:teuthology.orchestra.run.smithi064.stderr:dumped monmap epoch 3 2024-04-23T18:06:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:57 smithi032 ceph-mon[29221]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.154 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T18:06:58.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.a calling monitor election 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.c calling monitor election 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: monmap epoch 3 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: min_mon_release 19 (squid) 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: election_strategy: 1 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: fsmap 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mgrmap e13: x(active, since 60s) 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: overall HEALTH_OK 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:58.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:06:58.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:58.413 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:06:58.815 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mon.b calling monitor election 2024-04-23T18:06:58.815 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mon.b calling monitor election 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mon.c calling monitor election 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mon.a calling monitor election 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: monmap epoch 3 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:58.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: min_mon_release 19 (squid) 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: election_strategy: 1 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: fsmap 2024-04-23T18:06:58.817 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: mgrmap e13: x(active, since 62s) 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.818 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:58.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:06:58.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:58.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.b calling monitor election 2024-04-23T18:06:58.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.b calling monitor election 2024-04-23T18:06:58.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.c calling monitor election 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.a calling monitor election 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: monmap epoch 3 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: min_mon_release 19 (squid) 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: election_strategy: 1 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: fsmap 2024-04-23T18:06:58.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: mgrmap e13: x(active, since 62s) 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: overall HEALTH_OK 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:58.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:06:58.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mon.b calling monitor election 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mon.b calling monitor election 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mon.c calling monitor election 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mon.a calling monitor election 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: monmap epoch 3 2024-04-23T18:06:58.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: last_changed 2024-04-23T18:06:51.031323+0000 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: created 2024-04-23T18:04:48.613148+0000 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: min_mon_release 19 (squid) 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: election_strategy: 1 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: 1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: fsmap 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: mgrmap e13: x(active, since 62s) 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.924 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:06:58.925 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:06:59.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:59 smithi032 ceph-mon[29221]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:59.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:59 smithi032 ceph-mon[29221]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:06:59.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:59 smithi032 ceph-mon[29221]: Reconfiguring daemon mon.a on smithi032 2024-04-23T18:06:59.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:06:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:59.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:59 smithi040 ceph-mon[32626]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:59.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:59 smithi040 ceph-mon[32626]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:06:59.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:59 smithi040 ceph-mon[32626]: Reconfiguring daemon mon.a on smithi032 2024-04-23T18:06:59.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:06:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:06:59.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:59 smithi064 ceph-mon[33062]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:06:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:59 smithi064 ceph-mon[33062]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:06:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:59 smithi064 ceph-mon[33062]: Reconfiguring daemon mon.a on smithi032 2024-04-23T18:06:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:06:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:07:00.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- 172.21.15.32:0/1325465427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1084e0 msgr2=0x7f792c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1325465427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1084e0 0x7f792c1088c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f792400b0a0 tx=0x7f792400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- 172.21.15.32:0/1325465427 shutdown_connections 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1325465427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1084e0 0x7f792c1088c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- 172.21.15.32:0/1325465427 >> 172.21.15.32:0/1325465427 conn(0x7f792c076ea0 msgr2=0x7f792c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- 172.21.15.32:0/1325465427 shutdown_connections 2024-04-23T18:07:00.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- 172.21.15.32:0/1325465427 wait complete. 2024-04-23T18:07:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 Processor -- start 2024-04-23T18:07:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.379+0000 7f7932dc9700 1 -- start start 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f792c1084e0 0x7f792c101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 0x7f792c101e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f792c10c8d0 con 0x7f792c1a4350 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f792c10c750 con 0x7f792c1084e0 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7932dc9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f792c10ca50 con 0x7f792c1019a0 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57570/0 (socket says 172.21.15.32:57570) 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 -- 172.21.15.32:0/1487416885 learned_addr learned my addr 172.21.15.32:0/1487416885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7930b65700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f792c1084e0 0x7f792c101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f792bfff700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 0x7f792c101e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 -- 172.21.15.32:0/1487416885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 msgr2=0x7f792c101e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 0x7f792c101e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 -- 172.21.15.32:0/1487416885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f792c1084e0 msgr2=0x7f792c101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f792c1084e0 0x7f792c101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 -- 172.21.15.32:0/1487416885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792c1a6e70 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f792bfff700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 0x7f792c101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7931366700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b550 tx=0x7f791c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.380+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c015020 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.381+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f791c011ce0 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.381+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792c1a7160 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.381+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c010db0 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.381+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792c1a76a0 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.381+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f791c018020 con 0x7f792c1a4350 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.382+0000 7f7929ffb700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 0x7f791403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.382+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f792c0640a0 con 0x7f792c1a4350 2024-04-23T18:07:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.382+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f791c04c240 con 0x7f792c1a4350 2024-04-23T18:07:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.382+0000 7f7930b65700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 0x7f791403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.382+0000 7f7930b65700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 0x7f791403bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79240062a0 tx=0x7f7924005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:00.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.387+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f791c01f020 con 0x7f792c1a4350 2024-04-23T18:07:00.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.599+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f792c1a7960 con 0x7f792c1a4350 2024-04-23T18:07:00.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.600+0000 7f7929ffb700 1 -- 172.21.15.32:0/1487416885 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f791c017db0 con 0x7f792c1a4350 2024-04-23T18:07:00.595 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:07:00.595 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-23T18:07:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:07:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 msgr2=0x7f791403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 0x7f791403bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79240062a0 tx=0x7f7924005040 comp rx=0 tx=0).stop 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 msgr2=0x7f792c1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b550 tx=0x7f791c00b860 comp rx=0 tx=0).stop 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 shutdown_connections 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792c1a4350 0x7f792c1a6740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7914039ae0 0x7f791403bfa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f792c1084e0 0x7f792c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 --2- 172.21.15.32:0/1487416885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f792c1019a0 0x7f792c101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 >> 172.21.15.32:0/1487416885 conn(0x7f792c076ea0 msgr2=0x7f792c100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 shutdown_connections 2024-04-23T18:07:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:00.602+0000 7f7932dc9700 1 -- 172.21.15.32:0/1487416885 wait complete. 2024-04-23T18:07:01.218 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T18:07:01.218 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:07:01.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:07:01.246 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:07:01.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:07:01.311 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:07:01.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:07:01.341 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:07:01.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:07:01.406 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:07:01.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:07:01.439 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:07:01.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:07:01.509 INFO:tasks.cephadm:Adding mgr.x on smithi032 2024-04-23T18:07:01.509 INFO:tasks.cephadm:Adding mgr.y on smithi040 2024-04-23T18:07:01.509 INFO:tasks.cephadm:Adding mgr.z on smithi064 2024-04-23T18:07:01.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi032=x;smithi040=y;smithi064=z' 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1487416885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: Reconfiguring daemon mon.b on smithi040 2024-04-23T18:07:01.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1487416885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: Reconfiguring daemon mon.b on smithi040 2024-04-23T18:07:01.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1487416885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: Reconfiguring daemon mon.b on smithi040 2024-04-23T18:07:01.915 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.408 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:03.408 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:03.409 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:03 smithi064 ceph-mon[33062]: Reconfiguring daemon mon.c on smithi064 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 -- 172.21.15.64:0/4143780379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec103070 msgr2=0x7f49ec105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 --2- 172.21.15.64:0/4143780379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec103070 0x7f49ec105460 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009910 tx=0x7f49dc009c20 comp rx=0 tx=0).stop 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 -- 172.21.15.64:0/4143780379 shutdown_connections 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 --2- 172.21.15.64:0/4143780379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec103070 0x7f49ec105460 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 -- 172.21.15.64:0/4143780379 >> 172.21.15.64:0/4143780379 conn(0x7f49ec0fca80 msgr2=0x7f49ec0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 -- 172.21.15.64:0/4143780379 shutdown_connections 2024-04-23T18:07:03.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 -- 172.21.15.64:0/4143780379 wait complete. 2024-04-23T18:07:03.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.436+0000 7f49f29da700 1 Processor -- start 2024-04-23T18:07:03.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 -- start start 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec19e850 0x7f49ec1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ec19ecd0 0x7f49ec1a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1084c0 con 0x7f49ec19e850 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec108340 con 0x7f49ec19ecd0 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f29da700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec108640 con 0x7f49ec103070 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53800/0 (socket says 172.21.15.64:53800) 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 -- 172.21.15.64:0/622913589 learned_addr learned my addr 172.21.15.64:0/622913589 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49eb7fe700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec19e850 0x7f49ec1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49f0f77700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ec19ecd0 0x7f49ec1a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 -- 172.21.15.64:0/622913589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ec19ecd0 msgr2=0x7f49ec1a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ec19ecd0 0x7f49ec1a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 -- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec19e850 msgr2=0x7f49ec1a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec19e850 0x7f49ec1a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.437+0000 7f49ebfff700 1 -- 172.21.15.64:0/622913589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1a6720 con 0x7f49ec103070 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49ebfff700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49dc005950 tx=0x7f49dc004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc026070 con 0x7f49ec103070 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1a6a10 con 0x7f49ec103070 2024-04-23T18:07:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49dc00b6f0 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc0038e0 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.438+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1a6fd0 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.439+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f49dc029070 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.439+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec0640a0 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.439+0000 7f49e97fa700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 0x7f49d403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.439+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f49dc04d710 con 0x7f49ec103070 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.439+0000 7f49eb7fe700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 0x7f49d403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.440+0000 7f49eb7fe700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 0x7f49d403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009ad0 tx=0x7f49e0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.445+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49dc013070 con 0x7f49ec103070 2024-04-23T18:07:03.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:03.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:03 smithi040 ceph-mon[32626]: Reconfiguring daemon mon.c on smithi064 2024-04-23T18:07:03.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.668+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi040=y;smithi064=z", "target": ["mon-mgr", ""]}) -- 0x7f49ec062f10 con 0x7f49d4039ae0 2024-04-23T18:07:03.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.674+0000 7f49e97fa700 1 -- 172.21.15.64:0/622913589 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f49ec062f10 con 0x7f49d4039ae0 2024-04-23T18:07:03.669 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled mgr update... 2024-04-23T18:07:03.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.677+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 msgr2=0x7f49d403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.677+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 0x7f49d403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009ad0 tx=0x7f49e0009210 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 msgr2=0x7f49ec19e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49dc005950 tx=0x7f49dc004f80 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 shutdown_connections 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f49d4039ae0 0x7f49d403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ec19ecd0 0x7f49ec1a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49ec103070 0x7f49ec19e310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 --2- 172.21.15.64:0/622913589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ec19e850 0x7f49ec1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 >> 172.21.15.64:0/622913589 conn(0x7f49ec0fca80 msgr2=0x7f49ec0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 shutdown_connections 2024-04-23T18:07:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:03.678+0000 7f49f29da700 1 -- 172.21.15.64:0/622913589 wait complete. 2024-04-23T18:07:03.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:03.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:03 smithi032 ceph-mon[29221]: Reconfiguring daemon mon.c on smithi064 2024-04-23T18:07:04.563 DEBUG:teuthology.orchestra.run.smithi040:mgr.y> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T18:07:04.565 DEBUG:teuthology.orchestra.run.smithi064:mgr.z> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.z.service 2024-04-23T18:07:04.568 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T18:07:04.568 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:07:04.568 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:07:04.589 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:07:04.590 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-23 18:06:08.303754289 +0000 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-23 17:59:57.723497634 +0000 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-23 17:59:57.723497634 +0000 2024-04-23T18:07:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-23 17:59:57.723497634 +0000 2024-04-23T18:07:04.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:07:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-23T18:07:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-23T18:07:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000112312 s, 4.6 MB/s 2024-04-23T18:07:04.717 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:07:04.776 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-23T18:07:04.834 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:07:04.834 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:04.834 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T18:07:04.834 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:04.835 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:04.835 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-23 18:06:08.331753712 +0000 2024-04-23T18:07:04.835 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-23 17:59:58.073490384 +0000 2024-04-23T18:07:04.835 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-23 17:59:58.073490384 +0000 2024-04-23T18:07:04.835 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-23 17:59:58.073490384 +0000 2024-04-23T18:07:04.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:07:04.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:04 smithi040 ceph-mon[32626]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi040=y;smithi064=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:04.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:04 smithi040 ceph-mon[32626]: Saving service mgr spec with placement smithi032=x;smithi040=y;smithi064=z;count:3 2024-04-23T18:07:04.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:04 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-23T18:07:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-23T18:07:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000117106 s, 4.4 MB/s 2024-04-23T18:07:04.901 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:07:04.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:04 smithi064 ceph-mon[33062]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi040=y;smithi064=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:04.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:04 smithi064 ceph-mon[33062]: Saving service mgr spec with placement smithi032=x;smithi040=y;smithi064=z;count:3 2024-04-23T18:07:04.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:04 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:04.959 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-23T18:07:05.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:04 smithi032 ceph-mon[29221]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi040=y;smithi064=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:05.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:04 smithi032 ceph-mon[29221]: Saving service mgr spec with placement smithi032=x;smithi040=y;smithi064=z;count:3 2024-04-23T18:07:05.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:04 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-23 18:06:08.357753176 +0000 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-23 17:59:58.420483197 +0000 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-23 17:59:58.420483197 +0000 2024-04-23T18:07:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-23 17:59:58.420483197 +0000 2024-04-23T18:07:05.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:07:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-23T18:07:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-23T18:07:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.7422e-05 s, 5.3 MB/s 2024-04-23T18:07:05.083 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:07:05.140 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.197 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-23 18:06:08.383752640 +0000 2024-04-23T18:07:05.198 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-23 17:59:58.752476320 +0000 2024-04-23T18:07:05.198 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-23 17:59:58.752476320 +0000 2024-04-23T18:07:05.198 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-23 17:59:58.752476320 +0000 2024-04-23T18:07:05.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:07:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-23T18:07:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-23T18:07:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000109621 s, 4.7 MB/s 2024-04-23T18:07:05.262 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:07:05.319 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-23T18:07:05.320 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:07:05.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:07:05.340 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-23 18:06:50.375794885 +0000 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-23 18:00:27.075463791 +0000 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-23 18:00:27.075463791 +0000 2024-04-23T18:07:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-23 18:00:27.075463791 +0000 2024-04-23T18:07:05.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:07:05.467 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-23T18:07:05.467 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-23T18:07:05.467 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000287824 s, 1.8 MB/s 2024-04-23T18:07:05.468 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:07:05.529 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-23 18:06:50.375794885 +0000 2024-04-23T18:07:05.587 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-23 18:00:27.441458696 +0000 2024-04-23T18:07:05.588 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-23 18:00:27.441458696 +0000 2024-04-23T18:07:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-23 18:00:27.441458696 +0000 2024-04-23T18:07:05.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:07:05.656 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-23T18:07:05.656 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-23T18:07:05.656 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.00126961 s, 403 kB/s 2024-04-23T18:07:05.658 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:07:05.717 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-23T18:07:05.777 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:07:05.777 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.777 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-23T18:07:05.777 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.777 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.778 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-23 18:06:50.375794885 +0000 2024-04-23T18:07:05.778 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-23 18:00:27.778454004 +0000 2024-04-23T18:07:05.778 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-23 18:00:27.778454004 +0000 2024-04-23T18:07:05.778 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-23 18:00:27.778454004 +0000 2024-04-23T18:07:05.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:07:05.842 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:05.842 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:05.842 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:05.843 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:05.844 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-23T18:07:05.845 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-23T18:07:05.845 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000209056 s, 2.4 MB/s 2024-04-23T18:07:05.846 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:07:05.907 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-23T18:07:05.966 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:07:05.967 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:05.967 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-23T18:07:05.967 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:05.967 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:05.968 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-23 18:06:50.376794871 +0000 2024-04-23T18:07:05.968 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-23 18:00:28.133449062 +0000 2024-04-23T18:07:05.968 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-23 18:00:28.133449062 +0000 2024-04-23T18:07:05.968 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-23 18:00:28.133449062 +0000 2024-04-23T18:07:05.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:06.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:06.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:06.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-23T18:07:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-23T18:07:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000221738 s, 2.3 MB/s 2024-04-23T18:07:06.036 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:07:06.097 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-23T18:07:06.097 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:07:06.113 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:07:06.114 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 658 Links: 1 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-23 18:06:40.678442723 +0000 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-23 18:00:22.613932939 +0000 2024-04-23T18:07:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-23 18:00:22.613932939 +0000 2024-04-23T18:07:06.171 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-23 18:00:22.613932939 +0000 2024-04-23T18:07:06.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:06.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:06.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-23T18:07:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-23T18:07:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000253479 s, 2.0 MB/s 2024-04-23T18:07:06.203 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:07:06.260 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 678 Links: 1 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:06.315 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-23 18:06:40.678442723 +0000 2024-04-23T18:07:06.316 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-23 18:00:22.955926276 +0000 2024-04-23T18:07:06.316 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-23 18:00:22.955926276 +0000 2024-04-23T18:07:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-23 18:00:22.955926276 +0000 2024-04-23T18:07:06.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:07:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-23T18:07:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-23T18:07:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000175528 s, 2.9 MB/s 2024-04-23T18:07:06.380 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:07:06.436 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-04-23T18:07:06.491 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 696 Links: 1 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-23 18:06:40.679442704 +0000 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-23 18:00:23.303919496 +0000 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-23 18:00:23.303919496 +0000 2024-04-23T18:07:06.492 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-23 18:00:23.303919496 +0000 2024-04-23T18:07:06.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:07:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-23T18:07:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-23T18:07:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000235914 s, 2.2 MB/s 2024-04-23T18:07:06.555 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:07:06.612 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 715 Links: 1 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-23 18:06:40.679442704 +0000 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-23 18:00:23.658912580 +0000 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-23 18:00:23.658912580 +0000 2024-04-23T18:07:06.667 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-23 18:00:23.658912580 +0000 2024-04-23T18:07:06.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:07:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:06 smithi064 ceph-mon[33062]: Deploying daemon mgr.z on smithi064 2024-04-23T18:07:06.731 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-23T18:07:06.731 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-23T18:07:06.731 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00019112 s, 2.7 MB/s 2024-04-23T18:07:06.732 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:07:06.789 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-23T18:07:06.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:07:07.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:06 smithi032 ceph-mon[29221]: Deploying daemon mgr.z on smithi064 2024-04-23T18:07:07.019 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:06 smithi040 ceph-mon[32626]: Deploying daemon mgr.z on smithi064 2024-04-23T18:07:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:07:07.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:07 smithi032 ceph-mon[29221]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:07.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:07 smithi064 ceph-mon[33062]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:07.923 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:07 smithi064 systemd[1]: Starting Ceph mgr.z for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:07:08.031 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:07 smithi040 ceph-mon[32626]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 podman[34906]: 2024-04-23 18:07:08.02314154 +0000 UTC m=+0.101266176 container create c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 podman[34906]: 2024-04-23 18:07:07.94049508 +0000 UTC m=+0.018619684 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 podman[34906]: 2024-04-23 18:07:08.2337699 +0000 UTC m=+0.311894495 container init c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 podman[34906]: 2024-04-23 18:07:08.23660523 +0000 UTC m=+0.314729835 container start c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 ceph-mgr[34925]: -- 172.21.15.64:0/139133224 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564db9b545a0 con 0x564db9b29000 2024-04-23T18:07:08.294 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 bash[34906]: c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c 2024-04-23T18:07:08.595 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 systemd[1]: Started Ceph mgr.z for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:07:08.596 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:08.409+0000 7fc67f1d6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:07:08.596 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:08.600+0000 7fc67f1d6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:08.922 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:08.706+0000 7fc67f1d6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00949096 s, 1.1 GB/s 2024-04-23T18:07:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-23T18:07:09.374 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:09.026+0000 7fc67f1d6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:07:09.375 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:09.114+0000 7fc67f1d6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:07:09.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:09.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:09 smithi032 ceph-mon[29221]: Deploying daemon mgr.y on smithi040 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:09.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:09 smithi064 ceph-mon[33062]: Deploying daemon mgr.y on smithi040 2024-04-23T18:07:09.771 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:07:09.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:09.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:09.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:09 smithi040 ceph-mon[32626]: Deploying daemon mgr.y on smithi040 2024-04-23T18:07:10.618 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 systemd[1]: Starting Ceph mgr.y for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:07:10.672 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:10.378+0000 7fc67f1d6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:07:10.673 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:10.554+0000 7fc67f1d6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 podman[33997]: 2024-04-23 18:07:10.541735776 +0000 UTC m=+0.018247142 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 podman[33997]: 2024-04-23 18:07:10.640099259 +0000 UTC m=+0.116610598 container create 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 podman[33997]: 2024-04-23 18:07:10.826651552 +0000 UTC m=+0.303162891 container init 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 podman[33997]: 2024-04-23 18:07:10.829262593 +0000 UTC m=+0.305773938 container start 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 ceph-mgr[34016]: -- 172.21.15.40:0/2149054064 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c0a76dc5a0 con 0x55c0a76b0800 2024-04-23T18:07:10.896 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 bash[33997]: 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 2024-04-23T18:07:11.022 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:10.730+0000 7fc67f1d6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:07:11.022 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:11.027+0000 7fc67f1d6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:07:11.192 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:10 smithi040 systemd[1]: Started Ceph mgr.y for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:07:11.192 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:11.004+0000 7f5f1a213200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:07:11.192 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:11.196+0000 7f5f1a213200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:11.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 -- 172.21.15.32:0/2338976844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10103cc0 msgr2=0x7fbf10104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 --2- 172.21.15.32:0/2338976844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10103cc0 0x7fbf10104120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf040099c0 tx=0x7fbf04009cd0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 -- 172.21.15.32:0/2338976844 shutdown_connections 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 --2- 172.21.15.32:0/2338976844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10104660 0x7fbf1010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 --2- 172.21.15.32:0/2338976844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 0x7fbf10102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 --2- 172.21.15.32:0/2338976844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10103cc0 0x7fbf10104120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 -- 172.21.15.32:0/2338976844 >> 172.21.15.32:0/2338976844 conn(0x7fbf100fe060 msgr2=0x7fbf101004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 -- 172.21.15.32:0/2338976844 shutdown_connections 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.315+0000 7fbf16dba700 1 -- 172.21.15.32:0/2338976844 wait complete. 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 Processor -- start 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 -- start start 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 0x7fbf10198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 0x7fbf101991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010e5a0 con 0x7fbf10104660 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010e420 con 0x7fbf10103cc0 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf16dba700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010e720 con 0x7fbf10102ac0 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59194/0 (socket says 172.21.15.32:59194) 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 0x7fbf101991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 -- 172.21.15.32:0/3281579893 learned_addr learned my addr 172.21.15.32:0/3281579893 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf14b56700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 0x7fbf10198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 -- 172.21.15.32:0/3281579893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 msgr2=0x7fbf10198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 0x7fbf10198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 -- 172.21.15.32:0/3281579893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 msgr2=0x7fbf101991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 0x7fbf101991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf15357700 1 -- 172.21.15.32:0/3281579893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf101a08f0 con 0x7fbf10104660 2024-04-23T18:07:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.316+0000 7fbf0ffff700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 0x7fbf101991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf15357700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007ee0 tx=0x7fbf0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800dd20 con 0x7fbf10104660 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf08004020 con 0x7fbf10104660 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf101a0be0 con 0x7fbf10104660 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08015660 con 0x7fbf10104660 2024-04-23T18:07:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.317+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf101a10f0 con 0x7fbf10104660 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbf08015800 con 0x7fbf10104660 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf0dffb700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 0x7fbef803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbf0804c320 con 0x7fbf10104660 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf10069730 con 0x7fbf10104660 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf14b56700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 0x7fbef803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.318+0000 7fbf14b56700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 0x7fbef803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf10199b50 tx=0x7fbf0000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:11.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.323+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf08010070 con 0x7fbf10104660 2024-04-23T18:07:11.332 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:11.126+0000 7fc67f1d6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:07:11.332 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:11.337+0000 7fc67f1d6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:07:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:11.544+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fbf10062e80 con 0x7fbef8039890 2024-04-23T18:07:11.617 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:11.304+0000 7f5f1a213200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:11.672 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:11.542+0000 7fc67f1d6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:11.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:12.001 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:11.637+0000 7f5f1a213200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:07:12.001 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:11.735+0000 7f5f1a213200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:07:12.381 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:12.187+0000 7fc67f1d6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:07:12.557 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:12 smithi032 ceph-mon[29221]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:12.557 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:12.557 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:12.557 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:12.636 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-mon[33062]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:12.636 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:12.636 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:12.636 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:12 smithi040 ceph-mon[32626]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:12.922 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:12.641+0000 7fc67f1d6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:07:12.923 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:12.739+0000 7fc67f1d6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:07:13.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.036+0000 7f5f1a213200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:07:13.368 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.211+0000 7f5f1a213200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:13.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:12.948+0000 7fc67f1d6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:07:13.679 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.387+0000 7f5f1a213200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:07:13.679 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.684+0000 7f5f1a213200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-mon[32626]: Reconfiguring daemon mgr.x on smithi032 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:13 smithi032 ceph-mon[29221]: Reconfiguring daemon mgr.x on smithi032 2024-04-23T18:07:13.834 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:13.494+0000 7fc67f1d6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:07:13.834 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:13.589+0000 7fc67f1d6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:07:13.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:13.835 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-mon[33062]: Reconfiguring daemon mgr.x on smithi032 2024-04-23T18:07:13.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.779+0000 7f5f1a213200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:07:13.984 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:13.988+0000 7f5f1a213200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:07:14.172 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:13.839+0000 7fc67f1d6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:07:14.172 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:13.976+0000 7fc67f1d6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:07:14.173 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:14.073+0000 7fc67f1d6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:07:14.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:14.187+0000 7f5f1a213200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:14.455 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:14.193+0000 7fc67f1d6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:14.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:14 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]: dispatch 2024-04-23T18:07:14.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:14 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]': finished 2024-04-23T18:07:14.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:14 smithi032 ceph-mon[29221]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:07:14.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:14.805 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]: dispatch 2024-04-23T18:07:14.805 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]': finished 2024-04-23T18:07:14.805 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-mon[32626]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:07:14.805 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:14.922 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:14.506+0000 7fc67f1d6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:07:14.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]: dispatch 2024-04-23T18:07:14.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3390313473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a71e039-95f2-441c-86d2-65a3c98560e4"}]': finished 2024-04-23T18:07:14.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-mon[33062]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:07:14.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:15.117 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:14.810+0000 7f5f1a213200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:07:15.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:15.132+0000 7fc67f1d6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:07:15.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:07:15.354+0000 7fc67f1d6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:07:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:15.466+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fbf080148c0 con 0x7fbf10104660 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:15.245+0000 7f5f1a213200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:15.344+0000 7f5f1a213200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:15.551+0000 7f5f1a213200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2832616256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: Standby manager daemon z started 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:07:15.547 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:15 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2832616256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: Standby manager daemon z started 2024-04-23T18:07:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:07:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:15.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:07:15.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:15 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:15.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2832616256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: Standby manager daemon z started 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:07:15.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:15 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/3850327401' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:16.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.102+0000 7f5f1a213200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:07:16.368 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.196+0000 7f5f1a213200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:07:16.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:16 smithi032 ceph-mon[29221]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-23T18:07:16.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:07:16.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-mon[32626]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-23T18:07:16.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:07:16.795 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.445+0000 7f5f1a213200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:07:16.795 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.582+0000 7f5f1a213200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:07:16.795 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.680+0000 7f5f1a213200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:07:16.795 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:16.799+0000 7f5f1a213200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:16.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:16 smithi064 ceph-mon[33062]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-23T18:07:16.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:07:17.470 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:17.129+0000 7f5f1a213200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:07:17.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:17 smithi040 ceph-mon[32626]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:17.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:17 smithi032 ceph-mon[29221]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:17.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:17 smithi064 ceph-mon[33062]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:18.118 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:17.752+0000 7f5f1a213200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:07:18.118 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:07:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:07:17.974+0000 7f5f1a213200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:07:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:18.486+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fbf08021120 con 0x7fbf10104660 2024-04-23T18:07:18.664 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:18 smithi032 ceph-mon[29221]: Standby manager daemon y started 2024-04-23T18:07:18.664 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:18 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:07:18.664 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:18 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:18.664 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:18 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:07:18.664 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:18 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:18.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:18 smithi040 ceph-mon[32626]: Standby manager daemon y started 2024-04-23T18:07:18.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:18 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:07:18.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:18 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:18.759 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:18 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:07:18.759 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:18 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:18.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:18 smithi064 ceph-mon[33062]: Standby manager daemon y started 2024-04-23T18:07:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:18 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:07:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:18 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:07:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:18 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:07:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:18 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/378922612' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:07:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:19 smithi032 ceph-mon[29221]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:19 smithi032 ceph-mon[29221]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-23T18:07:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:19 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:07:19.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:19 smithi040 ceph-mon[32626]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:19.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:19 smithi040 ceph-mon[32626]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-23T18:07:19.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:19 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:07:19.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:19 smithi064 ceph-mon[33062]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:19 smithi064 ceph-mon[33062]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-23T18:07:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:19 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:07:20.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:20.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:20.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:20.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:20.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:20.992 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:21.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:21.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:21.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:21.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:21.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:21 smithi032 ceph-mon[29221]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:22.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:21 smithi040 ceph-mon[32626]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:22.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:21 smithi064 ceph-mon[33062]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:22.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:07:22.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:23.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:07:23.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:23.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:07:23.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:23.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:23 smithi032 ceph-mon[29221]: Deploying daemon osd.0 on smithi032 2024-04-23T18:07:23.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:23 smithi032 ceph-mon[29221]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:24.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:23 smithi040 ceph-mon[32626]: Deploying daemon osd.0 on smithi032 2024-04-23T18:07:24.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:23 smithi040 ceph-mon[32626]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:24.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:23 smithi064 ceph-mon[33062]: Deploying daemon osd.0 on smithi032 2024-04-23T18:07:24.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:23 smithi064 ceph-mon[33062]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:26.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:25 smithi032 ceph-mon[29221]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:26.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:25 smithi040 ceph-mon[32626]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:26.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:25 smithi064 ceph-mon[33062]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:27 smithi032 ceph-mon[29221]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:27 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:27 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:27 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:28.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:27 smithi040 ceph-mon[32626]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:28.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:27 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:27 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:27 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:28.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:27 smithi064 ceph-mon[33062]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:28.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:27 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:27 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:28.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:27 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:30.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:29 smithi032 ceph-mon[29221]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:30.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:29 smithi040 ceph-mon[32626]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:30.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:29 smithi064 ceph-mon[33062]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:30.350 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-04-23T18:07:30.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.355+0000 7fbf0dffb700 1 -- 172.21.15.32:0/3281579893 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf10062e80 con 0x7fbef8039890 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 msgr2=0x7fbef803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 0x7fbef803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf10199b50 tx=0x7fbf0000b410 comp rx=0 tx=0).stop 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 msgr2=0x7fbf101a0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007ee0 tx=0x7fbf0800d3b0 comp rx=0 tx=0).stop 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 shutdown_connections 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf10104660 0x7fbf101a0220 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf10102ac0 0x7fbf10198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbef8039890 0x7fbef803bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 --2- 172.21.15.32:0/3281579893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10103cc0 0x7fbf101991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 >> 172.21.15.32:0/3281579893 conn(0x7fbf100fe060 msgr2=0x7fbf10107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 shutdown_connections 2024-04-23T18:07:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:07:30.358+0000 7fbf16dba700 1 -- 172.21.15.32:0/3281579893 wait complete. 2024-04-23T18:07:30.932 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T18:07:30.974 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/vg_nvme/lv_4... 2024-04-23T18:07:30.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:07:31.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:31.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:31 smithi032 ceph-mon[29221]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:07:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:07:31.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:31.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:31 smithi040 ceph-mon[32626]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:07:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:31 smithi064 ceph-mon[33062]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:07:32.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:32 smithi040 ceph-mon[32626]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:07:32.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:32 smithi040 ceph-mon[32626]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:07:32.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:32 smithi040 ceph-mon[32626]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:07:32.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-23T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-23T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126864 s, 827 MB/s 2024-04-23T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-04-23T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-23T18:07:33.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:32 smithi064 ceph-mon[33062]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:07:33.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:32 smithi064 ceph-mon[33062]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:07:33.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:32 smithi064 ceph-mon[33062]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:07:33.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:33.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:32 smithi032 ceph-mon[29221]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:07:33.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:32 smithi032 ceph-mon[29221]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:07:33.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:32 smithi032 ceph-mon[29221]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:07:33.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:33.508 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0[43724]: 2024-04-23T18:07:33.249+0000 7f4e1f0b7700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T18:07:33.509 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0[43724]: 2024-04-23T18:07:33.252+0000 7f4e16ea2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:07:33.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-04-23T18:07:34.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:34.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:07:34.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:07:34.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:33 smithi032 ceph-mon[29221]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' 2024-04-23T18:07:34.017 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:07:34.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:34.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:07:34.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:07:34.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:33 smithi040 ceph-mon[32626]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' 2024-04-23T18:07:34.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:34.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:07:34.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:07:34.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:34.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:33 smithi064 ceph-mon[33062]: from='osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420]' entity='osd.0' 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.028 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:35.029 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:35.029 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.029 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] boot 2024-04-23T18:07:35.029 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:07:35.029 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] boot 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:07:35.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: osd.0 [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] boot 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:07:35.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:07:35.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 -- 172.21.15.40:0/1778611144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0102ac0 msgr2=0x7fa0d0102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 --2- 172.21.15.40:0/1778611144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0102ee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009990 tx=0x7fa0c0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 -- 172.21.15.40:0/1778611144 shutdown_connections 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 --2- 172.21.15.40:0/1778611144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0104660 0x7fa0d010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 --2- 172.21.15.40:0/1778611144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0102ee0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 --2- 172.21.15.40:0/1778611144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 -- 172.21.15.40:0/1778611144 >> 172.21.15.40:0/1778611144 conn(0x7fa0d00fe060 msgr2=0x7fa0d01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 -- 172.21.15.40:0/1778611144 shutdown_connections 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 -- 172.21.15.40:0/1778611144 wait complete. 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.529+0000 7fa0d6117700 1 Processor -- start 2024-04-23T18:07:35.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 -- start start 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0104660 0x7fa0d01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010e5a0 con 0x7fa0d0102ac0 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010e420 con 0x7fa0d0103cc0 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0d6117700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010e720 con 0x7fa0d0104660 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:35.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47782/0 (socket says 172.21.15.40:47782) 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 -- 172.21.15.40:0/499417487 learned_addr learned my addr 172.21.15.40:0/499417487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0cffff700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0104660 0x7fa0d01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0cf7fe700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 -- 172.21.15.40:0/499417487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0104660 msgr2=0x7fa0d01a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0104660 0x7fa0d01a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 -- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 msgr2=0x7fa0d0198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0ceffd700 1 -- 172.21.15.40:0/499417487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d01a08a0 con 0x7fa0d0103cc0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.530+0000 7fa0cf7fe700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0ceffd700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400e9a0 tx=0x7fa0c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c400cbd0 con 0x7fa0d0103cc0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c400cd70 con 0x7fa0d0103cc0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01a0b90 con 0x7fa0d0103cc0 2024-04-23T18:07:35.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4018680 con 0x7fa0d0103cc0 2024-04-23T18:07:35.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.531+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01a1120 con 0x7fa0d0103cc0 2024-04-23T18:07:35.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.532+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d0069730 con 0x7fa0d0103cc0 2024-04-23T18:07:35.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.533+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa0c4010430 con 0x7fa0d0103cc0 2024-04-23T18:07:35.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.534+0000 7fa0ccff9700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 0x7fa0b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:35.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.534+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7fa0c40cefc0 con 0x7fa0d0103cc0 2024-04-23T18:07:35.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.534+0000 7fa0cf7fe700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 0x7fa0b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:35.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.534+0000 7fa0cf7fe700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 0x7fa0b80a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0000c00 tx=0x7fa0c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:35.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.538+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0c409d410 con 0x7fa0d0103cc0 2024-04-23T18:07:35.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:35.754+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa0d0062e80 con 0x7fa0b80a3c30 2024-04-23T18:07:36.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:36.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi032 2024-04-23T18:07:36.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-23T18:07:36.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:36.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:36.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:36.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:36.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi032 2024-04-23T18:07:36.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-23T18:07:36.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:36.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:36.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:36.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:07:36.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi032 2024-04-23T18:07:36.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-23T18:07:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:07:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:07:36.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:37.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:36 smithi040 ceph-mon[32626]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:37.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:36 smithi040 ceph-mon[32626]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:07:37.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:36 smithi064 ceph-mon[33062]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:37.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:36 smithi064 ceph-mon[33062]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:07:37.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:36 smithi032 ceph-mon[29221]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:37.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:36 smithi032 ceph-mon[29221]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:07:38.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:37 smithi040 ceph-mon[32626]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:38.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:37 smithi064 ceph-mon[33062]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:38.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:37 smithi032 ceph-mon[29221]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:39.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:38 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/3509776046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:38 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:38 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]': finished 2024-04-23T18:07:39.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:38 smithi040 ceph-mon[32626]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:07:39.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:39.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:38 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/3509776046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:38 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:38 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]': finished 2024-04-23T18:07:39.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:38 smithi064 ceph-mon[33062]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:07:39.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:39.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:38 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/3509776046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:38 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]: dispatch 2024-04-23T18:07:39.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:38 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "780165d2-56a4-4fee-98a0-bc98ba8c9ef7"}]': finished 2024-04-23T18:07:39.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:38 smithi032 ceph-mon[29221]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:07:39.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:40.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:39 smithi040 ceph-mon[32626]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:40.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:39 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/1864787655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:40.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:39 smithi064 ceph-mon[33062]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:40.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:39 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/1864787655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:40.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:39 smithi032 ceph-mon[29221]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:40.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:39 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/1864787655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:07:42.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:41 smithi040 ceph-mon[32626]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:41 smithi064 ceph-mon[33062]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:42.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:41 smithi032 ceph-mon[29221]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:44.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:43 smithi040 ceph-mon[32626]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:44.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:43 smithi064 ceph-mon[33062]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:44.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:43 smithi032 ceph-mon[29221]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:46.101 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:45 smithi040 ceph-mon[32626]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:46.101 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:45 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:07:46.101 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:45 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:46.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:45 smithi064 ceph-mon[33062]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:46.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:45 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:07:46.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:45 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:46.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:45 smithi032 ceph-mon[29221]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:46.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:45 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:07:46.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:45 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:46.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:46 smithi040 ceph-mon[32626]: Deploying daemon osd.1 on smithi040 2024-04-23T18:07:47.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:46 smithi064 ceph-mon[33062]: Deploying daemon osd.1 on smithi040 2024-04-23T18:07:47.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:46 smithi032 ceph-mon[29221]: Deploying daemon osd.1 on smithi040 2024-04-23T18:07:48.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:47 smithi040 ceph-mon[32626]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:47 smithi064 ceph-mon[33062]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:48.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:47 smithi032 ceph-mon[29221]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:49.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:49 smithi040 ceph-mon[32626]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:50.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:49 smithi064 ceph-mon[33062]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:49 smithi032 ceph-mon[29221]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:51.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:51 smithi032 ceph-mon[29221]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:51.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:51 smithi040 ceph-mon[32626]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:51.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:51 smithi064 ceph-mon[33062]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:51.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:51.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:53.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:53.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:53.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:53.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:53.846 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.917+0000 7fa0ccff9700 1 -- 172.21.15.40:0/499417487 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa0d0062e80 con 0x7fa0b80a3c30 2024-04-23T18:07:53.915 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2024-04-23T18:07:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 msgr2=0x7fa0b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 0x7fa0b80a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0000c00 tx=0x7fa0c0005fb0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 msgr2=0x7fa0d0199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400e9a0 tx=0x7fa0c400ecb0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 shutdown_connections 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0d0104660 0x7fa0d01a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0d0102ac0 0x7fa0d0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa0b80a3c30 0x7fa0b80a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 --2- 172.21.15.40:0/499417487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0103cc0 0x7fa0d0199190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.919+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 >> 172.21.15.40:0/499417487 conn(0x7fa0d00fe060 msgr2=0x7fa0d0107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.920+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 shutdown_connections 2024-04-23T18:07:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:07:53.920+0000 7fa0d6117700 1 -- 172.21.15.40:0/499417487 wait complete. 2024-04-23T18:07:53.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:53.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:53.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:53.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:53.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:54.562 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T18:07:54.564 INFO:tasks.cephadm:Deploying osd.2 on smithi064 with /dev/vg_nvme/lv_4... 2024-04-23T18:07:54.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:07:54.819 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:07:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:54 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:54 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:54 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:54 smithi040 ceph-mon[32626]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:54 smithi040 ceph-mon[32626]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:55.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:54 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:54 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:54 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:55.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:54 smithi064 ceph-mon[33062]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:55.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:54 smithi064 ceph-mon[33062]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:54 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:54 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:54 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:54 smithi032 ceph-mon[29221]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:54 smithi032 ceph-mon[29221]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:07:56.010 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-mon[32626]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.011 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1[37238]: 2024-04-23T18:07:55.904+0000 7fc4fd14d700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T18:07:56.011 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:07:55 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1[37238]: 2024-04-23T18:07:55.909+0000 7fc4f8f40700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:07:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:07:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:07:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:55 smithi064 ceph-mon[33062]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:07:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:07:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: from='osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:55 smithi032 ceph-mon[29221]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112367 s, 933 MB/s 2024-04-23T18:07:56.835 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-23T18:07:57.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:56 smithi040 ceph-mon[32626]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:07:57.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:56 smithi040 ceph-mon[32626]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:07:57.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.156 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:56 smithi064 ceph-mon[33062]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:07:57.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:56 smithi064 ceph-mon[33062]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:07:57.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.157 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:56 smithi032 ceph-mon[29221]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:07:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:56 smithi032 ceph-mon[29221]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:07:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:57.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-04-23T18:07:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:07:58.160 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:07:58.160 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] boot 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi040 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:58.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:07:58.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] boot 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi040 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:58.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: osd.1 [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] boot 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi040 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-23T18:07:58.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:07:58.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:07:59.133 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:58 smithi064 ceph-mon[33062]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:07:59.133 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:58 smithi064 ceph-mon[33062]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 -- 172.21.15.64:0/2685047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404104640 msgr2=0x7f740410aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 --2- 172.21.15.64:0/2685047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404104640 0x7f740410aef0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00b0a0 tx=0x7f73fc00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 -- 172.21.15.64:0/2685047848 shutdown_connections 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 --2- 172.21.15.64:0/2685047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404104640 0x7f740410aef0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 --2- 172.21.15.64:0/2685047848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404102aa0 0x7f7404102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 --2- 172.21.15.64:0/2685047848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 0x7f7404104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 -- 172.21.15.64:0/2685047848 >> 172.21.15.64:0/2685047848 conn(0x7f74040fe040 msgr2=0x7f7404100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:59.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 -- 172.21.15.64:0/2685047848 shutdown_connections 2024-04-23T18:07:59.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.244+0000 7f740a459700 1 -- 172.21.15.64:0/2685047848 wait complete. 2024-04-23T18:07:59.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 Processor -- start 2024-04-23T18:07:59.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 -- start start 2024-04-23T18:07:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 0x7f7404074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404104640 0x7f7404075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f740410e590 con 0x7f7404104640 2024-04-23T18:07:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f740410e410 con 0x7f7404103ca0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f740a459700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f740410e710 con 0x7f7404102aa0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49590/0 (socket says 172.21.15.64:49590) 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 -- 172.21.15.64:0/1591430092 learned_addr learned my addr 172.21.15.64:0/1591430092 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f74037fe700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 0x7f7404074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f74089f6700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404104640 0x7f7404075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 -- 172.21.15.64:0/1591430092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 msgr2=0x7f7404074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 0x7f7404074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 -- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404104640 msgr2=0x7f7404075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404104640 0x7f7404075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.245+0000 7f7403fff700 1 -- 172.21.15.64:0/1591430092 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7404075c20 con 0x7f7404102aa0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f7403fff700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73f400e970 tx=0x7f73f400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4004220 con 0x7f7404102aa0 2024-04-23T18:07:59.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f40043c0 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74041a2d60 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4003ce0 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.246+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74041a31e0 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.247+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7404069730 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.248+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f73f400c7e0 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.249+0000 7f74017fa700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 0x7f73ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.249+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f73f40ced80 con 0x7f7404102aa0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.249+0000 7f74037fe700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 0x7f73ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.249+0000 7f74037fe700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 0x7f73ec0a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f73f8004630 tx=0x7f73f800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:59.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.252+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73f409d040 con 0x7f7404102aa0 2024-04-23T18:07:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:58 smithi032 ceph-mon[29221]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:07:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:58 smithi032 ceph-mon[29221]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:07:59.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:58 smithi040 ceph-mon[32626]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:07:59.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:58 smithi040 ceph-mon[32626]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:07:59.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:07:59.474+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7404062e80 con 0x7f73ec0a3c30 2024-04-23T18:08:00.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:59 smithi064 ceph-mon[33062]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:00.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:00.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:00.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:07:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:59 smithi032 ceph-mon[29221]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:07:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:00.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:59 smithi040 ceph-mon[32626]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:00.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:00.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:00.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:07:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:01.247 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:00 smithi064 ceph-mon[33062]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:01.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:00 smithi032 ceph-mon[29221]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:01.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:00 smithi040 ceph-mon[32626]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/1453640767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]': finished 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:03.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/574839702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/1453640767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]': finished 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/574839702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/1453640767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]: dispatch 2024-04-23T18:08:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d71493-93bf-4dfe-ac4b-6a378dd2e244"}]': finished 2024-04-23T18:08:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:08:03.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:03.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/574839702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:05 smithi040 ceph-mon[32626]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:05 smithi064 ceph-mon[33062]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:05 smithi032 ceph-mon[29221]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:07.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:07 smithi064 ceph-mon[33062]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:07.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:07 smithi040 ceph-mon[32626]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:07.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:07 smithi032 ceph-mon[29221]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:09.351 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:09 smithi064 ceph-mon[33062]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:09 smithi040 ceph-mon[32626]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:09 smithi032 ceph-mon[29221]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:10.337 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:10 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:08:10.337 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:10 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:10.337 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:10 smithi064 ceph-mon[33062]: Deploying daemon osd.2 on smithi064 2024-04-23T18:08:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:10 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:08:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:10 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:10.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:10 smithi040 ceph-mon[32626]: Deploying daemon osd.2 on smithi064 2024-04-23T18:08:10.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:10 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:08:10.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:10 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:10.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:10 smithi032 ceph-mon[29221]: Deploying daemon osd.2 on smithi064 2024-04-23T18:08:11.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:11 smithi064 ceph-mon[33062]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:11.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:11 smithi040 ceph-mon[32626]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:11.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:11 smithi032 ceph-mon[29221]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:13.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:13 smithi064 ceph-mon[33062]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:13 smithi040 ceph-mon[32626]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:13.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:13 smithi032 ceph-mon[29221]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:15.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:15 smithi064 ceph-mon[33062]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:15.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.349 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:15 smithi040 ceph-mon[32626]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:15 smithi032 ceph-mon[29221]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:15.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:17.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:17 smithi064 ceph-mon[33062]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:17.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:17 smithi040 ceph-mon[32626]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:17 smithi032 ceph-mon[29221]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:18.340 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 2 on host 'smithi064' 2024-04-23T18:08:18.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.345+0000 7f74017fa700 1 -- 172.21.15.64:0/1591430092 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7404062e80 con 0x7f73ec0a3c30 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 msgr2=0x7f73ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 0x7f73ec0a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f73f8004630 tx=0x7f73f800b3c0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 msgr2=0x7f74040767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73f400e970 tx=0x7f73f400ec80 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 shutdown_connections 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7404104640 0x7f7404075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7404102aa0 0x7f74040767e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73ec0a3c30 0x7f73ec0a60f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 --2- 172.21.15.64:0/1591430092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404103ca0 0x7f7404074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 >> 172.21.15.64:0/1591430092 conn(0x7f74040fe040 msgr2=0x7f74041070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 shutdown_connections 2024-04-23T18:08:18.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:08:18.348+0000 7f740a459700 1 -- 172.21.15.64:0/1591430092 wait complete. 2024-04-23T18:08:18.937 DEBUG:teuthology.orchestra.run.smithi064:osd.2> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T18:08:18.940 INFO:tasks.cephadm:Deploying osd.3 on smithi032 with /dev/vg_nvme/lv_3... 2024-04-23T18:08:18.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:08:19.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:19.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:18 smithi064 ceph-mon[33062]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:19.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:18 smithi032 ceph-mon[29221]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:19.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:19.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:18 smithi040 ceph-mon[32626]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:08:20.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-mon[33062]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:08:20.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-mon[33062]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:08:20.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:20.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-mon[33062]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:20.384 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-mon[33062]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:20.568 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:20 smithi032 ceph-mon[29221]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:08:20.568 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:20 smithi032 ceph-mon[29221]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:08:20.568 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:20.568 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:20 smithi032 ceph-mon[29221]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:20.568 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:20 smithi032 ceph-mon[29221]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:20.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:20 smithi040 ceph-mon[32626]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:08:20.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:20 smithi040 ceph-mon[32626]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:08:20.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:20.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:20 smithi040 ceph-mon[32626]: from='osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:20.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:20 smithi040 ceph-mon[32626]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:08:21.173 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2[38185]: 2024-04-23T18:08:20.806+0000 7fa12efdc700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T18:08:21.173 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2[38185]: 2024-04-23T18:08:20.810+0000 7fa1265c6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:08:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:08:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:08:21.273 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-23T18:08:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-23T18:08:21.273 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0098789 s, 1.1 GB/s 2024-04-23T18:08:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-23T18:08:21.583 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:21.584 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:08:21.584 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:08:21.584 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.584 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.584 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:21 smithi064 ceph-mon[33062]: from='osd.2 ' entity='osd.2' 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:21 smithi032 ceph-mon[29221]: from='osd.2 ' entity='osd.2' 2024-04-23T18:08:21.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:21.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:08:21.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:08:21.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:21.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:21 smithi040 ceph-mon[32626]: from='osd.2 ' entity='osd.2' 2024-04-23T18:08:21.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-04-23T18:08:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:22.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:08:22.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:08:22.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] boot 2024-04-23T18:08:22.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:08:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:08:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] boot 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: osd.2 [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] boot 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:08:22.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:22.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:23.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.740+0000 7faa00119700 1 -- 172.21.15.32:0/2754823135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8104640 msgr2=0x7fa9f810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.740+0000 7faa00119700 1 --2- 172.21.15.32:0/2754823135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8104640 0x7fa9f810aef0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000b0d0 tx=0x7fa9f000b3e0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- 172.21.15.32:0/2754823135 shutdown_connections 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- 172.21.15.32:0/2754823135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8104640 0x7fa9f810aef0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- 172.21.15.32:0/2754823135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8102aa0 0x7fa9f8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- 172.21.15.32:0/2754823135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- 172.21.15.32:0/2754823135 >> 172.21.15.32:0/2754823135 conn(0x7fa9f80fe040 msgr2=0x7fa9f8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- 172.21.15.32:0/2754823135 shutdown_connections 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- 172.21.15.32:0/2754823135 wait complete. 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 Processor -- start 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- start start 2024-04-23T18:08:23.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8104640 0x7fa9f8075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.741+0000 7faa00119700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810e570 con 0x7fa9f8102aa0 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7faa00119700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810e3f0 con 0x7fa9f8104640 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7faa00119700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810e6f0 con 0x7fa9f8103ca0 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60184/0 (socket says 172.21.15.32:60184) 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 -- 172.21.15.32:0/4200377730 learned_addr learned my addr 172.21.15.32:0/4200377730 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fd6b4700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 -- 172.21.15.32:0/4200377730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 msgr2=0x7fa9f8074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:23.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 -- 172.21.15.32:0/4200377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8104640 msgr2=0x7fa9f8075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fe6b6700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8104640 0x7fa9f8075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8104640 0x7fa9f8075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 -- 172.21.15.32:0/4200377730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f8075c20 con 0x7fa9f8102aa0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fd6b4700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9fdeb5700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400d980 tx=0x7fa9f400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4017030 con 0x7fa9f8102aa0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f81a2cb0 con 0x7fa9f8102aa0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f40044d0 con 0x7fa9f8102aa0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4003aa0 con 0x7fa9f8102aa0 2024-04-23T18:08:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.742+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f81a31c0 con 0x7fa9f8102aa0 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.743+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f8069730 con 0x7fa9f8102aa0 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.745+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa9f401a070 con 0x7fa9f8102aa0 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.745+0000 7fa9eeffd700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 0x7fa9e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.745+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fa9f40ced00 con 0x7fa9f8102aa0 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.745+0000 7fa9fd6b4700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 0x7fa9e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:23.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.746+0000 7fa9fd6b4700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 0x7fa9e40a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005fd0 tx=0x7fa9e8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.749+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9f409cba0 con 0x7fa9f8102aa0 2024-04-23T18:08:23.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:23.991+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa9f8062e80 con 0x7fa9e40a3bb0 2024-04-23T18:08:24.067 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42255]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.067 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42255]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.067 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42255]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.067 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42255]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi064 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:08:24.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:23 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi064 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:08:24.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:23 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:24.117 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48300]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48300]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48303]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48303]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48303]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.118 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:08:23 smithi032 sudo[48303]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi064 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:08:24.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:23 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:08:24.119 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:23 smithi040 sudo[41070]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.119 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:23 smithi040 sudo[41070]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.119 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:23 smithi040 sudo[41070]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.119 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41070]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.120 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41073]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.120 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41073]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.120 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41073]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.120 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41073]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.422 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42258]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.423 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42258]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.423 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42258]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.423 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42258]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42261]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42261]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42261]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42261]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42264]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42264]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42264]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 sudo[42264]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48373]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48373]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48373]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48373]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48376]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48376]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 sudo[48376]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41076]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:08:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41076]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41076]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41076]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:08:24.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 sudo[41079]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:08:24.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:24.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:08:24.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:24.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:24.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:24.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:24.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:24 smithi032 ceph-mon[29221]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:24.988+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa9f4098b10 con 0x7fa9f8102aa0 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.120 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.120 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:24 smithi040 ceph-mon[32626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:08:25.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:24 smithi064 ceph-mon[33062]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:08:25.935 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:25 smithi032 ceph-mon[29221]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:25.935 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:25 smithi032 ceph-mon[29221]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:25.935 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:25 smithi032 ceph-mon[29221]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:08:25.935 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:25 smithi032 ceph-mon[29221]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:08:26.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:25 smithi040 ceph-mon[32626]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:26.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:25 smithi040 ceph-mon[32626]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:26.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:25 smithi040 ceph-mon[32626]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:08:26.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:25 smithi040 ceph-mon[32626]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:08:26.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:25 smithi064 ceph-mon[33062]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:26.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:25 smithi064 ceph-mon[33062]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:26.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:25 smithi064 ceph-mon[33062]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:08:26.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:25 smithi064 ceph-mon[33062]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:08:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:26 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]: dispatch 2024-04-23T18:08:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:26 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]': finished 2024-04-23T18:08:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:26 smithi040 ceph-mon[32626]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:08:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:26 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:26 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]: dispatch 2024-04-23T18:08:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:26 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]': finished 2024-04-23T18:08:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:26 smithi064 ceph-mon[33062]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:08:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:26 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:27.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:26 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]: dispatch 2024-04-23T18:08:27.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:26 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3461150352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "253ddac0-f39a-459c-bef1-2d3bd28a2b2f"}]': finished 2024-04-23T18:08:27.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:26 smithi032 ceph-mon[29221]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:08:27.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:26 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:28.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:27 smithi040 ceph-mon[32626]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:28.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:27 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2903582421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:28.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:27 smithi064 ceph-mon[33062]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:28.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:27 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2903582421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:28.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:27 smithi032 ceph-mon[29221]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:28.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:27 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2903582421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:30.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:29 smithi040 ceph-mon[32626]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:30.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:29 smithi064 ceph-mon[33062]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:30.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:29 smithi032 ceph-mon[29221]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:31.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:31 smithi032 ceph-mon[29221]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:32.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:31 smithi040 ceph-mon[32626]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:32.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:31 smithi064 ceph-mon[33062]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:34.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:33 smithi032 ceph-mon[29221]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:34.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:33 smithi040 ceph-mon[32626]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:34.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:33 smithi064 ceph-mon[33062]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:35.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:08:35.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:35.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:34 smithi040 ceph-mon[32626]: Deploying daemon osd.3 on smithi032 2024-04-23T18:08:35.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:08:35.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:35.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:34 smithi032 ceph-mon[29221]: Deploying daemon osd.3 on smithi032 2024-04-23T18:08:35.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:08:35.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:35.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:34 smithi064 ceph-mon[33062]: Deploying daemon osd.3 on smithi032 2024-04-23T18:08:36.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:35 smithi032 ceph-mon[29221]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:36.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:35 smithi040 ceph-mon[32626]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:36.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:35 smithi064 ceph-mon[33062]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:38.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:37 smithi040 ceph-mon[32626]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:38.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:37 smithi064 ceph-mon[33062]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:38.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:37 smithi032 ceph-mon[29221]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:40.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:39 smithi032 ceph-mon[29221]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:40.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:40.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:39 smithi040 ceph-mon[32626]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:40.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:40.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:39 smithi064 ceph-mon[33062]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:40.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:40.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:42.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:41 smithi040 ceph-mon[32626]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:41 smithi064 ceph-mon[33062]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:42.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:41 smithi032 ceph-mon[29221]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:42.247 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 3 on host 'smithi032' 2024-04-23T18:08:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.252+0000 7fa9eeffd700 1 -- 172.21.15.32:0/4200377730 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa9f8062e80 con 0x7fa9e40a3bb0 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 msgr2=0x7fa9e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 0x7fa9e40a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005fd0 tx=0x7fa9e8005e30 comp rx=0 tx=0).stop 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 msgr2=0x7fa9f80767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400d980 tx=0x7fa9f400dd40 comp rx=0 tx=0).stop 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 shutdown_connections 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa9e40a3bb0 0x7fa9e40a6070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9f8104640 0x7fa9f8075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9f8102aa0 0x7fa9f80767e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 --2- 172.21.15.32:0/4200377730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9f8103ca0 0x7fa9f8074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 >> 172.21.15.32:0/4200377730 conn(0x7fa9f80fe040 msgr2=0x7fa9f8106f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 shutdown_connections 2024-04-23T18:08:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:08:42.255+0000 7faa00119700 1 -- 172.21.15.32:0/4200377730 wait complete. 2024-04-23T18:08:43.220 DEBUG:teuthology.orchestra.run.smithi032:osd.3> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T18:08:43.223 INFO:tasks.cephadm:Deploying osd.4 on smithi040 with /dev/vg_nvme/lv_3... 2024-04-23T18:08:43.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:08:43.486 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:08:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:43 smithi032 ceph-mon[29221]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:43 smithi032 ceph-mon[29221]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:08:43.578 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:43 smithi040 ceph-mon[32626]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:43.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:43 smithi040 ceph-mon[32626]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:08:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:43 smithi064 ceph-mon[33062]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:43.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:43 smithi064 ceph-mon[33062]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:08:44.705 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:08:44.747 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:44.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:44.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:08:44.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:08:44.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:08:44.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:45.008 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:08:44.868+0000 7f50b5b5b700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T18:08:45.008 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:08:44 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:08:44.874+0000 7f50b194e700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109614 s, 957 MB/s 2024-04-23T18:08:45.354 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-04-23T18:08:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] boot 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:45 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:45.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-04-23T18:08:46.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:46.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:08:46.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:08:46.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] boot 2024-04-23T18:08:46.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:08:46.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:45 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: from='osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: osd.3 [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] boot 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:08:46.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:45 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:08:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:08:47.090 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:46 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:08:47.090 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:46 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:08:47.090 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:46 smithi040 ceph-mon[32626]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:08:47.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:46 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:08:47.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:46 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:08:47.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:46 smithi064 ceph-mon[33062]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:08:47.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:46 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:08:47.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:46 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:08:47.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:46 smithi032 ceph-mon[29221]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:08:47.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1242939517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38102bc0 msgr2=0x7fdb38103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1242939517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38102bc0 0x7fdb38103020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdb240099c0 tx=0x7fdb24009cd0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1242939517 shutdown_connections 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1242939517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38103560 0x7fdb38109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1242939517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 0x7fdb38101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1242939517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38102bc0 0x7fdb38103020 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1242939517 >> 172.21.15.40:0/1242939517 conn(0x7fdb380fcf60 msgr2=0x7fdb380ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1242939517 shutdown_connections 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.943+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1242939517 wait complete. 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 Processor -- start 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 -- start start 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 0x7fdb38105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38102bc0 0x7fdb38103fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810d4b0 con 0x7fdb381019c0 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810d330 con 0x7fdb38103560 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3eac3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810d630 con 0x7fdb38102bc0 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3d060700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3d060700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36428/0 (socket says 172.21.15.40:36428) 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3d060700 1 -- 172.21.15.40:0/1504340937 learned_addr learned my addr 172.21.15.40:0/1504340937 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.944+0000 7fdb3c85f700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 0x7fdb38105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb37fff700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38102bc0 0x7fdb38103fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 -- 172.21.15.40:0/1504340937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38102bc0 msgr2=0x7fdb38103fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38102bc0 0x7fdb38103fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 -- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 msgr2=0x7fdb38105900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 0x7fdb38105900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 -- 172.21.15.40:0/1504340937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb38104db0 con 0x7fdb38103560 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3d060700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b970 tx=0x7fdb3000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000d860 con 0x7fdb38103560 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb3000e430 con 0x7fdb38103560 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb381a2d80 con 0x7fdb38103560 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb30017660 con 0x7fdb38103560 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.945+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb381a3180 con 0x7fdb38103560 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.946+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb38069730 con 0x7fdb38103560 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.947+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdb3000da00 con 0x7fdb38103560 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.948+0000 7fdb35ffb700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 0x7fdb280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.948+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7fdb300cee10 con 0x7fdb38103560 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.948+0000 7fdb3c85f700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 0x7fdb280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.949+0000 7fdb3c85f700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 0x7fdb280a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb38102a20 tx=0x7fdb2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:47.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:47.953+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb30013030 con 0x7fdb38103560 2024-04-23T18:08:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:08:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi032 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.119 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:47 smithi040 ceph-mon[32626]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:08:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:08:48.173+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdb380ff150 con 0x7fdb280a3c40 2024-04-23T18:08:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:08:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi032 2024-04-23T18:08:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:47 smithi064 ceph-mon[33062]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:08:48.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:08:48.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi032 2024-04-23T18:08:48.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:08:48.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:47 smithi032 ceph-mon[29221]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:08:49.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:49.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:49.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:49.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:49.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:49.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:49.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:08:49.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:08:49.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:49 smithi040 ceph-mon[32626]: pgmap v103: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:49 smithi040 ceph-mon[32626]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:49 smithi040 ceph-mon[32626]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-23T18:08:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:49 smithi040 ceph-mon[32626]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:08:50.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:49 smithi064 ceph-mon[33062]: pgmap v103: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:49 smithi064 ceph-mon[33062]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:49 smithi064 ceph-mon[33062]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-23T18:08:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:49 smithi064 ceph-mon[33062]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:08:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:49 smithi032 ceph-mon[29221]: pgmap v103: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:49 smithi032 ceph-mon[29221]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:49 smithi032 ceph-mon[29221]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-23T18:08:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:49 smithi032 ceph-mon[29221]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:08:51.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:50 smithi064 ceph-mon[33062]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:08:51.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:50 smithi032 ceph-mon[29221]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:08:51.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:50 smithi040 ceph-mon[32626]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:08:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/2832781632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]': finished 2024-04-23T18:08:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:08:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:08:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:51 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/883799486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/2832781632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]': finished 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:08:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:08:52.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:51 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/883799486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/2832781632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]: dispatch 2024-04-23T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f898fa-45b8-4ebd-bf04-b5758d43461e"}]': finished 2024-04-23T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:51 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/883799486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:08:54.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:53 smithi064 ceph-mon[33062]: pgmap v108: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:54.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:53 smithi032 ceph-mon[29221]: pgmap v108: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:54.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:53 smithi040 ceph-mon[32626]: pgmap v108: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-23T18:08:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:54 smithi040 ceph-mon[32626]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:08:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:54 smithi040 ceph-mon[32626]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-23T18:08:55.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:54 smithi040 ceph-mon[32626]: Cluster is now healthy 2024-04-23T18:08:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:54 smithi032 ceph-mon[29221]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:08:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:54 smithi032 ceph-mon[29221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-23T18:08:55.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:54 smithi032 ceph-mon[29221]: Cluster is now healthy 2024-04-23T18:08:55.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:54 smithi064 ceph-mon[33062]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:08:55.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:54 smithi064 ceph-mon[33062]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-23T18:08:55.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:54 smithi064 ceph-mon[33062]: Cluster is now healthy 2024-04-23T18:08:57.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:57 smithi040 ceph-mon[32626]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-23T18:08:57.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:57 smithi064 ceph-mon[33062]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-23T18:08:57.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:57 smithi032 ceph-mon[29221]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-23T18:08:59.321 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:59 smithi040 ceph-mon[32626]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-23T18:08:59.321 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:08:59.321 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:59.321 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:08:59 smithi040 ceph-mon[32626]: Deploying daemon osd.4 on smithi040 2024-04-23T18:08:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:59 smithi064 ceph-mon[33062]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-23T18:08:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:08:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:08:59 smithi064 ceph-mon[33062]: Deploying daemon osd.4 on smithi040 2024-04-23T18:08:59.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:59 smithi032 ceph-mon[29221]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-23T18:08:59.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:08:59.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:59.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:08:59 smithi032 ceph-mon[29221]: Deploying daemon osd.4 on smithi040 2024-04-23T18:09:01.324 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:01 smithi040 ceph-mon[32626]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:09:01.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:01 smithi064 ceph-mon[33062]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:09:01.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:01 smithi032 ceph-mon[29221]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:09:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:03 smithi040 ceph-mon[32626]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:09:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:03 smithi064 ceph-mon[33062]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:09:03.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:03 smithi032 ceph-mon[29221]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:09:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:05 smithi032 ceph-mon[29221]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:05.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:05 smithi064 ceph-mon[33062]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:05.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.516 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:05.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:05 smithi040 ceph-mon[32626]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:05.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:05.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:07.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:07 smithi040 ceph-mon[32626]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:07.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:07 smithi032 ceph-mon[29221]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:07 smithi064 ceph-mon[33062]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:08.090 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 4 on host 'smithi040' 2024-04-23T18:09:08.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.095+0000 7fdb35ffb700 1 -- 172.21.15.40:0/1504340937 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdb380ff150 con 0x7fdb280a3c40 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 msgr2=0x7fdb280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 0x7fdb280a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb38102a20 tx=0x7fdb2c00b410 comp rx=0 tx=0).stop 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 msgr2=0x7fdb381044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b970 tx=0x7fdb3000bc80 comp rx=0 tx=0).stop 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 shutdown_connections 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdb280a3c40 0x7fdb280a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb38103560 0x7fdb381044e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb381019c0 0x7fdb38105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 --2- 172.21.15.40:0/1504340937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb38102bc0 0x7fdb38103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 >> 172.21.15.40:0/1504340937 conn(0x7fdb380fcf60 msgr2=0x7fdb380fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 shutdown_connections 2024-04-23T18:09:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:09:08.099+0000 7fdb3eac3700 1 -- 172.21.15.40:0/1504340937 wait complete. 2024-04-23T18:09:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:08 smithi040 ceph-mon[32626]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:08 smithi064 ceph-mon[33062]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:08.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:08.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:08.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:08.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:08 smithi032 ceph-mon[29221]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:09:08.879 DEBUG:teuthology.orchestra.run.smithi040:osd.4> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T18:09:08.881 INFO:tasks.cephadm:Deploying osd.5 on smithi064 with /dev/vg_nvme/lv_3... 2024-04-23T18:09:08.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:09:09.157 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:09.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:09 smithi064 ceph-mon[33062]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:09:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:09:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:09 smithi040 ceph-mon[32626]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:09.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:09.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:09:09.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: from='osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:09.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:09:09.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:09.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:09 smithi032 ceph-mon[29221]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:09:10.868 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:09:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:09:10.536+0000 7fe4f0490700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T18:09:10.868 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:09:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:09:10.544+0000 7fe4e827b700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112201 s, 935 MB/s 2024-04-23T18:09:11.234 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.235 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:11 smithi064 ceph-mon[33062]: from='osd.4 ' entity='osd.4' 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:11 smithi040 ceph-mon[32626]: from='osd.4 ' entity='osd.4' 2024-04-23T18:09:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:09:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:09:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:11.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:11 smithi032 ceph-mon[29221]: from='osd.4 ' entity='osd.4' 2024-04-23T18:09:12.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-04-23T18:09:12.249 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] boot 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] boot 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:09:12.372 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:12.373 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:09:12.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:09:12.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: osd.4 [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] boot 2024-04-23T18:09:12.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:12.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:13 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi040 2024-04-23T18:09:13.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:13 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-23T18:09:13.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:13 smithi040 ceph-mon[32626]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:13.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:13 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi040 2024-04-23T18:09:13.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:13 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-23T18:09:13.386 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:13 smithi064 ceph-mon[33062]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:13.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:13 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi040 2024-04-23T18:09:13.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:13 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-23T18:09:13.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:13 smithi032 ceph-mon[29221]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:09:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 -- 172.21.15.64:0/1576285729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 msgr2=0x7f715810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 --2- 172.21.15.64:0/1576285729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f715810aef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f715000b0a0 tx=0x7f715000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 -- 172.21.15.64:0/1576285729 shutdown_connections 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 --2- 172.21.15.64:0/1576285729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f715810aef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 --2- 172.21.15.64:0/1576285729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158102aa0 0x7f7158102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 --2- 172.21.15.64:0/1576285729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158103ca0 0x7f7158104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 -- 172.21.15.64:0/1576285729 >> 172.21.15.64:0/1576285729 conn(0x7f71580fe040 msgr2=0x7f7158100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 -- 172.21.15.64:0/1576285729 shutdown_connections 2024-04-23T18:09:13.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 -- 172.21.15.64:0/1576285729 wait complete. 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.854+0000 7f715f03b700 1 Processor -- start 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 -- start start 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158102aa0 0x7f7158198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158103ca0 0x7f71581990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f715810e540 con 0x7f7158103ca0 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f715810e3c0 con 0x7f7158102aa0 2024-04-23T18:09:13.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715f03b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f715810e6c0 con 0x7f7158104640 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43466/0 (socket says 172.21.15.64:43466) 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 -- 172.21.15.64:0/3639405662 learned_addr learned my addr 172.21.15.64:0/3639405662 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715cdd7700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158102aa0 0x7f7158198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f7157fff700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158103ca0 0x7f71581990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 -- 172.21.15.64:0/3639405662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158102aa0 msgr2=0x7f7158198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158102aa0 0x7f7158198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 -- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158103ca0 msgr2=0x7f71581990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158103ca0 0x7f71581990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.855+0000 7f715d5d8700 1 -- 172.21.15.64:0/3639405662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7158074cf0 con 0x7f7158104640 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f715d5d8700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f715000b070 tx=0x7f7150004d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150021070 con 0x7f7158104640 2024-04-23T18:09:13.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7150007b80 con 0x7f7158104640 2024-04-23T18:09:13.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7158074f80 con 0x7f7158104640 2024-04-23T18:09:13.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150004790 con 0x7f7158104640 2024-04-23T18:09:13.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.856+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71580754b0 con 0x7f7158104640 2024-04-23T18:09:13.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.857+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158069730 con 0x7f7158104640 2024-04-23T18:09:13.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.858+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7150003a00 con 0x7f7158104640 2024-04-23T18:09:13.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.859+0000 7f7155ffb700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 0x7f71480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:13.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.859+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f715000f020 con 0x7f7158104640 2024-04-23T18:09:13.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.859+0000 7f715cdd7700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 0x7f71480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:13.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.859+0000 7f715cdd7700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 0x7f71480a6180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7158103b00 tx=0x7f714c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:13.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:13.864+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f715002a020 con 0x7f7158104640 2024-04-23T18:09:14.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:14.082+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7158109970 con 0x7f71480a3cc0 2024-04-23T18:09:14.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:14 smithi040 ceph-mon[32626]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:09:14.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:14.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:14.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:14 smithi064 ceph-mon[33062]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:09:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:14.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:14 smithi032 ceph-mon[29221]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:09:14.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:14.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:14.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:15 smithi040 ceph-mon[32626]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:15.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:15 smithi040 ceph-mon[32626]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:15 smithi064 ceph-mon[33062]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:15 smithi064 ceph-mon[33062]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:15 smithi032 ceph-mon[29221]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:15 smithi032 ceph-mon[29221]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:17.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:17.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/3891402247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]': finished 2024-04-23T18:09:17.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:09:17.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:17 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/3891402247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]': finished 2024-04-23T18:09:17.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:09:17.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:17 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/3891402247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]: dispatch 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9de82db1-b3dc-4181-9d27-fa440325720c"}]': finished 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:09:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:17 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:18.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:18 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/3483349206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:18.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:18 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/3483349206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:18.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:18 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/3483349206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:19.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:19 smithi064 ceph-mon[33062]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:19.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:19 smithi032 ceph-mon[29221]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:19.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:19 smithi040 ceph-mon[32626]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:21.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:21 smithi064 ceph-mon[33062]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:21.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:21 smithi032 ceph-mon[29221]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:21.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:21 smithi040 ceph-mon[32626]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:23.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:23 smithi064 ceph-mon[33062]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:23.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:23 smithi032 ceph-mon[29221]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:23 smithi040 ceph-mon[32626]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:25.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:25 smithi064 ceph-mon[33062]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:25.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:09:25.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.255 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:25 smithi064 ceph-mon[33062]: Deploying daemon osd.5 on smithi064 2024-04-23T18:09:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:25 smithi032 ceph-mon[29221]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:09:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:25 smithi032 ceph-mon[29221]: Deploying daemon osd.5 on smithi064 2024-04-23T18:09:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:25 smithi040 ceph-mon[32626]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:09:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:25 smithi040 ceph-mon[32626]: Deploying daemon osd.5 on smithi064 2024-04-23T18:09:27.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:27 smithi064 ceph-mon[33062]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:27 smithi032 ceph-mon[29221]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:27.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:27 smithi040 ceph-mon[32626]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:29 smithi064 ceph-mon[33062]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:29 smithi032 ceph-mon[29221]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:29.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:29 smithi040 ceph-mon[32626]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:31.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:30 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:30 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:30 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:31.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:30 smithi032 ceph-mon[29221]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:30 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:30 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:30 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:31.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:30 smithi040 ceph-mon[32626]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:30 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:30 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:30 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:31.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:30 smithi064 ceph-mon[33062]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:33 smithi040 ceph-mon[32626]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:33.380 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:33 smithi064 ceph-mon[33062]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:33.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.455+0000 7f7155ffb700 1 -- 172.21.15.64:0/3639405662 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7158109970 con 0x7f71480a3cc0 2024-04-23T18:09:33.450 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 5 on host 'smithi064' 2024-04-23T18:09:33.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 msgr2=0x7f71480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:33.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 0x7f71480a6180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7158103b00 tx=0x7f714c00b410 comp rx=0 tx=0).stop 2024-04-23T18:09:33.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 msgr2=0x7f71581a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:33.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f715000b070 tx=0x7f7150004d70 comp rx=0 tx=0).stop 2024-04-23T18:09:33.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 shutdown_connections 2024-04-23T18:09:33.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7158104640 0x7f71581a0150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:33.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f71480a3cc0 0x7f71480a6180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7158102aa0 0x7f7158198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 --2- 172.21.15.64:0/3639405662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7158103ca0 0x7f71581990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 >> 172.21.15.64:0/3639405662 conn(0x7f71580fe040 msgr2=0x7f7158106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 shutdown_connections 2024-04-23T18:09:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:09:33.458+0000 7f715f03b700 1 -- 172.21.15.64:0/3639405662 wait complete. 2024-04-23T18:09:33.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:33 smithi032 ceph-mon[29221]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:34.028 DEBUG:teuthology.orchestra.run.smithi064:osd.5> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T18:09:34.032 INFO:tasks.cephadm:Deploying osd.6 on smithi032 with /dev/vg_nvme/lv_2... 2024-04-23T18:09:34.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:09:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.667 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:09:34.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:34.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-mon[33062]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:35.687 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:35.688 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:09:35.688 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:09:35.688 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:35.688 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:35.688 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:35 smithi032 ceph-mon[29221]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:35.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:35.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:09:35.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:09:35.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:35.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: from='osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:35.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:35 smithi040 ceph-mon[32626]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:09:36.129 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:09:35.933+0000 7ff559eb6700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T18:09:36.129 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:09:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:09:35.939+0000 7ff5514a0700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00940251 s, 1.1 GB/s 2024-04-23T18:09:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:36 smithi032 ceph-mon[29221]: from='osd.5 ' entity='osd.5' 2024-04-23T18:09:36.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:36 smithi064 ceph-mon[33062]: from='osd.5 ' entity='osd.5' 2024-04-23T18:09:36.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:09:36.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:09:36.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:09:36.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:36.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:36 smithi040 ceph-mon[32626]: from='osd.5 ' entity='osd.5' 2024-04-23T18:09:36.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-04-23T18:09:37.249 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:37.624 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] boot 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:09:37.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] boot 2024-04-23T18:09:37.633 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:09:37.634 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:37 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:37.768 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: osd.5 [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] boot 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:09:37.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:37 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi064 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi064 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-23T18:09:38.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 -- 172.21.15.32:0/25879305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4104660 msgr2=0x7f00a410af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 --2- 172.21.15.32:0/25879305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4104660 0x7f00a410af10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f00940099c0 tx=0x7f0094009cd0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 -- 172.21.15.32:0/25879305 shutdown_connections 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 --2- 172.21.15.32:0/25879305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4104660 0x7f00a410af10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 --2- 172.21.15.32:0/25879305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 0x7f00a4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 --2- 172.21.15.32:0/25879305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4103cc0 0x7f00a4104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 -- 172.21.15.32:0/25879305 >> 172.21.15.32:0/25879305 conn(0x7f00a40fe060 msgr2=0x7f00a41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 -- 172.21.15.32:0/25879305 shutdown_connections 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 -- 172.21.15.32:0/25879305 wait complete. 2024-04-23T18:09:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.883+0000 7f00a91e6700 1 Processor -- start 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 -- start start 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 0x7f00a4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 0x7f00a41a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410e560 con 0x7f00a4103cc0 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410e3e0 con 0x7f00a4104660 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a91e6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410e6e0 con 0x7f00a4102ac0 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48852/0 (socket says 172.21.15.32:48852) 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 -- 172.21.15.32:0/3848850407 learned_addr learned my addr 172.21.15.32:0/3848850407 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a359e700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 0x7f00a41a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a2d9d700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 0x7f00a4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 -- 172.21.15.32:0/3848850407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 msgr2=0x7f00a4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 0x7f00a4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 -- 172.21.15.32:0/3848850407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 msgr2=0x7f00a41a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 0x7f00a41a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a259c700 1 -- 172.21.15.32:0/3848850407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a41a07b0 con 0x7f00a4103cc0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.884+0000 7f00a359e700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 0x7f00a41a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f00a259c700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f008c00e9d0 tx=0x7f008c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c00cbd0 con 0x7f00a4103cc0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f008c00cd70 con 0x7f00a4103cc0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a41a0aa0 con 0x7f00a4103cc0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c018680 con 0x7f00a4103cc0 2024-04-23T18:09:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.885+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a41a1030 con 0x7f00a4103cc0 2024-04-23T18:09:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.886+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a4069730 con 0x7f00a4103cc0 2024-04-23T18:09:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.887+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f008c010430 con 0x7f00a4103cc0 2024-04-23T18:09:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.888+0000 7f009bfff700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 0x7f00900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.888+0000 7f00a2d9d700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 0x7f00900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.888+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f008c0cf720 con 0x7f00a4103cc0 2024-04-23T18:09:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.888+0000 7f00a2d9d700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 0x7f00900a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f00a4103b20 tx=0x7f009c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:38.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:38.893+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f008c09cfa0 con 0x7f00a4103cc0 2024-04-23T18:09:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:09:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi064 2024-04-23T18:09:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:38.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:39.111+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f00a4062e80 con 0x7f00900a3cc0 2024-04-23T18:09:39.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:39 smithi032 ceph-mon[29221]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:39.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:39 smithi032 ceph-mon[29221]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:09:39.740 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:39.740 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:39.740 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:39 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:39.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:39 smithi040 ceph-mon[32626]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:39.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:39 smithi040 ceph-mon[32626]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:09:39.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:39.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:39.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:39 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:39.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:39 smithi064 ceph-mon[33062]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:39.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:39 smithi064 ceph-mon[33062]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:09:39.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:39.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:39.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:39 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:40.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:40 smithi032 ceph-mon[29221]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:40.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:40 smithi032 ceph-mon[29221]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:09:40.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:40 smithi040 ceph-mon[32626]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:40.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:40 smithi040 ceph-mon[32626]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:09:40.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:40 smithi064 ceph-mon[33062]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:40.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:40 smithi064 ceph-mon[33062]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:09:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:41 smithi032 ceph-mon[29221]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:41.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:41 smithi032 ceph-mon[29221]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:09:41.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:41 smithi040 ceph-mon[32626]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:41.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:41 smithi040 ceph-mon[32626]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:09:41.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:41 smithi064 ceph-mon[33062]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:41.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:41 smithi064 ceph-mon[33062]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:09:42.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:42 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]: dispatch 2024-04-23T18:09:42.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:42 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]': finished 2024-04-23T18:09:42.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:42 smithi032 ceph-mon[29221]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:09:42.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:42 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:09:42.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:42 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]: dispatch 2024-04-23T18:09:42.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:42 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]': finished 2024-04-23T18:09:42.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:42 smithi040 ceph-mon[32626]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:09:42.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:42 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:09:42.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:42 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]: dispatch 2024-04-23T18:09:42.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:42 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/476338700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea694844-38fa-4af5-842f-1d69ba8d0679"}]': finished 2024-04-23T18:09:42.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:42 smithi064 ceph-mon[33062]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:09:42.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:42 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:09:43.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:43 smithi032 ceph-mon[29221]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:43.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:43 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1190307017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:43.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:43 smithi040 ceph-mon[32626]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:43.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:43 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1190307017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:43.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:43 smithi064 ceph-mon[33062]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:43.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:43 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1190307017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:45.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:45 smithi032 ceph-mon[29221]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:09:45.800 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:45 smithi064 ceph-mon[33062]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:09:45.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:45 smithi040 ceph-mon[32626]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:09:47.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:47 smithi032 ceph-mon[29221]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:09:47.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:47 smithi040 ceph-mon[32626]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:09:47.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:47 smithi064 ceph-mon[33062]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:09:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:49 smithi032 ceph-mon[29221]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:09:49.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:49 smithi040 ceph-mon[32626]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:09:49.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:49 smithi064 ceph-mon[33062]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:09:50.633 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:50 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:09:50.634 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:50 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:50.634 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:50 smithi032 ceph-mon[29221]: Deploying daemon osd.6 on smithi032 2024-04-23T18:09:50.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:50 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:09:50.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:50 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:50.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:50 smithi040 ceph-mon[32626]: Deploying daemon osd.6 on smithi032 2024-04-23T18:09:50.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:50 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:09:50.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:50 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:50.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:50 smithi064 ceph-mon[33062]: Deploying daemon osd.6 on smithi032 2024-04-23T18:09:51.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:51 smithi032 ceph-mon[29221]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-23T18:09:51.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:51 smithi040 ceph-mon[32626]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-23T18:09:51.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:51 smithi064 ceph-mon[33062]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-23T18:09:53.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:53 smithi032 ceph-mon[29221]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T18:09:53.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:53 smithi040 ceph-mon[32626]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T18:09:53.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:53 smithi064 ceph-mon[33062]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T18:09:55.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:55 smithi032 ceph-mon[29221]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:55.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:55 smithi040 ceph-mon[32626]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:55.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:55 smithi064 ceph-mon[33062]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:09:56.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:57.676 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:57 smithi032 ceph-mon[29221]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:57 smithi040 ceph-mon[32626]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:57.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:57 smithi064 ceph-mon[33062]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:59.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.021+0000 7f009bfff700 1 -- 172.21.15.32:0/3848850407 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f00a4062e80 con 0x7f00900a3cc0 2024-04-23T18:09:59.016 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 6 on host 'smithi032' 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 msgr2=0x7f00900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 0x7f00900a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f00a4103b20 tx=0x7f009c00b040 comp rx=0 tx=0).stop 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 msgr2=0x7f00a4199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f008c00e9d0 tx=0x7f008c00ece0 comp rx=0 tx=0).stop 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 shutdown_connections 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00900a3cc0 0x7f00900a6180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a4104660 0x7f00a41a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00a4102ac0 0x7f00a4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 --2- 172.21.15.32:0/3848850407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a4103cc0 0x7f00a4199090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.022+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 >> 172.21.15.32:0/3848850407 conn(0x7f00a40fe060 msgr2=0x7f00a4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.023+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 shutdown_connections 2024-04-23T18:09:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:09:59.023+0000 7f00a91e6700 1 -- 172.21.15.32:0/3848850407 wait complete. 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.709 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:09:59 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.738 DEBUG:teuthology.orchestra.run.smithi032:osd.6> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T18:09:59.740 INFO:tasks.cephadm:Deploying osd.7 on smithi040 with /dev/vg_nvme/lv_2... 2024-04-23T18:09:59.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:09:59.768 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:59.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:09:59.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.769 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:09:59 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:09:59.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:09:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:09:59 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:09:59.996 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:10:01.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:10:01.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:10:01.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:10:01.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:01.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:10:01.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:00 smithi032 ceph-mon[29221]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: overall HEALTH_OK 2024-04-23T18:10:01.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:00 smithi040 ceph-mon[32626]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:01.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:10:01.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:10:01.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:10:01.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:01.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:10:01.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:00 smithi064 ceph-mon[33062]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:01.758 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:10:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:10:01.467+0000 7fa8ac7c6700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T18:10:01.759 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:10:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:10:01.480+0000 7fa8a45b1700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111435 s, 941 MB/s 2024-04-23T18:10:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-23T18:10:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:02 smithi032 ceph-mon[29221]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:10:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:02 smithi032 ceph-mon[29221]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:10:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:02 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:02 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:02 smithi032 ceph-mon[29221]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' 2024-04-23T18:10:02.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:02 smithi040 ceph-mon[32626]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:10:02.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:02 smithi040 ceph-mon[32626]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:10:02.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:02 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:02 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:02 smithi040 ceph-mon[32626]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' 2024-04-23T18:10:02.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:02 smithi064 ceph-mon[33062]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:10:02.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:02 smithi064 ceph-mon[33062]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:10:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:02 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:02 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:02.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:02 smithi064 ceph-mon[33062]: from='osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575]' entity='osd.6' 2024-04-23T18:10:03.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-04-23T18:10:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:10:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:10:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] boot 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi032 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:03.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] boot 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi032 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:03.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:10:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:10:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:10:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: osd.6 [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] boot 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi032 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:03.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:03.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:03.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:04 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:04 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:04 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:04 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:04.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:04 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:04.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:04 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:04.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:04 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:04.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:04 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:04.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:04 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:04.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:04 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:04.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:04 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:04.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:04 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1609203150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30103cc0 msgr2=0x7fca30104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1609203150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30103cc0 0x7fca30104120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca180099c0 tx=0x7fca18009cd0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1609203150 shutdown_connections 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1609203150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30104660 0x7fca3010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1609203150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 0x7fca30102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1609203150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30103cc0 0x7fca30104120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1609203150 >> 172.21.15.40:0/1609203150 conn(0x7fca300fe060 msgr2=0x7fca301004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1609203150 shutdown_connections 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1609203150 wait complete. 2024-04-23T18:10:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 Processor -- start 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 -- start start 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 0x7fca300767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 0x7fca30074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010e530 con 0x7fca30102ac0 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010e3b0 con 0x7fca30104660 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca34e3f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010e6b0 con 0x7fca30103cc0 2024-04-23T18:10:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca2ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca2ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47010/0 (socket says 172.21.15.40:47010) 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.053+0000 7fca2ed9d700 1 -- 172.21.15.40:0/1836067968 learned_addr learned my addr 172.21.15.40:0/1836067968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2e59c700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 0x7fca300767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2dd9b700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 0x7fca30074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 -- 172.21.15.40:0/1836067968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 msgr2=0x7fca30074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 0x7fca30074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 -- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 msgr2=0x7fca300767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 0x7fca300767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 -- 172.21.15.40:0/1836067968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca30075c20 con 0x7fca30104660 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2dd9b700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 0x7fca30074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca2ed9d700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c900 tx=0x7fca2800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28007b00 con 0x7fca30104660 2024-04-23T18:10:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca301a2cb0 con 0x7fca30104660 2024-04-23T18:10:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca2800ce50 con 0x7fca30104660 2024-04-23T18:10:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28018660 con 0x7fca30104660 2024-04-23T18:10:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.054+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca301a31c0 con 0x7fca30104660 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.055+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca30069730 con 0x7fca30104660 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.056+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fca2800f450 con 0x7fca30104660 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.057+0000 7fca277fe700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 0x7fca1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.057+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fca280cf640 con 0x7fca30104660 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.057+0000 7fca2e59c700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 0x7fca1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.058+0000 7fca2e59c700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 0x7fca1c0a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca30103b20 tx=0x7fca2000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.062+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca28014030 con 0x7fca30104660 2024-04-23T18:10:05.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:05 smithi040 ceph-mon[32626]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:05.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:05 smithi040 ceph-mon[32626]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:10:05.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:05.281+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fca30062e80 con 0x7fca1c0a3c40 2024-04-23T18:10:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:05 smithi064 ceph-mon[33062]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:05 smithi064 ceph-mon[33062]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:10:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:05 smithi032 ceph-mon[29221]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:05 smithi032 ceph-mon[29221]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:10:06.247 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:06 smithi040 ceph-mon[32626]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:06.247 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:06 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:06.247 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:06 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:06.247 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:06 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:06 smithi064 ceph-mon[33062]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:06 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:06 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:06 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:06.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:06 smithi032 ceph-mon[29221]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:06.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:06 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:06.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:06 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:06.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:06 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:07.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:07 smithi040 ceph-mon[32626]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:07.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:07 smithi064 ceph-mon[33062]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:07.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:07 smithi032 ceph-mon[29221]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/2764460492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]': finished 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:09.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:09 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/3427650746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/2764460492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]': finished 2024-04-23T18:10:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:10:09.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:09.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:09 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/3427650746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/2764460492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]: dispatch 2024-04-23T18:10:09.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6eaac83-e79b-4a1e-8ab6-31b047aca03b"}]': finished 2024-04-23T18:10:09.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:10:09.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:09.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:09 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/3427650746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:11.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:11 smithi040 ceph-mon[32626]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:11 smithi064 ceph-mon[33062]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:11.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:11 smithi032 ceph-mon[29221]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:13 smithi064 ceph-mon[33062]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:13.448 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:13 smithi040 ceph-mon[32626]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:13.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:13 smithi032 ceph-mon[29221]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:15.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:15 smithi040 ceph-mon[32626]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:15 smithi064 ceph-mon[33062]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:15.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:15 smithi032 ceph-mon[29221]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:10:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:16 smithi040 ceph-mon[32626]: Deploying daemon osd.7 on smithi040 2024-04-23T18:10:16.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:10:16.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:16.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:16 smithi064 ceph-mon[33062]: Deploying daemon osd.7 on smithi040 2024-04-23T18:10:16.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:10:16.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:16.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:16 smithi032 ceph-mon[29221]: Deploying daemon osd.7 on smithi040 2024-04-23T18:10:17.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:17 smithi040 ceph-mon[32626]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:17 smithi064 ceph-mon[33062]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:17 smithi032 ceph-mon[29221]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:19.420 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:19 smithi040 ceph-mon[32626]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:19.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:19 smithi064 ceph-mon[33062]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:19.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:19 smithi032 ceph-mon[29221]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:21.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:21 smithi040 ceph-mon[32626]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:21.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:21 smithi064 ceph-mon[33062]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:21.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:21 smithi032 ceph-mon[29221]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:22.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:22.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:22.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:23.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:23 smithi032 ceph-mon[29221]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:23 smithi040 ceph-mon[32626]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:23.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:23 smithi064 ceph-mon[33062]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:25.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.049+0000 7fca277fe700 1 -- 172.21.15.40:0/1836067968 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fca30062e80 con 0x7fca1c0a3c40 2024-04-23T18:10:25.044 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 7 on host 'smithi040' 2024-04-23T18:10:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.051+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 msgr2=0x7fca1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.051+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 0x7fca1c0a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca30103b20 tx=0x7fca2000c040 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 msgr2=0x7fca30075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c900 tx=0x7fca2800cc10 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 shutdown_connections 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca1c0a3c40 0x7fca1c0a6100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca30104660 0x7fca30075370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca30102ac0 0x7fca300767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 --2- 172.21.15.40:0/1836067968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca30103cc0 0x7fca30074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 >> 172.21.15.40:0/1836067968 conn(0x7fca300fe060 msgr2=0x7fca30106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 shutdown_connections 2024-04-23T18:10:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:10:25.052+0000 7fca34e3f700 1 -- 172.21.15.40:0/1836067968 wait complete. 2024-04-23T18:10:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.619 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:25 smithi040 ceph-mon[32626]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.655 DEBUG:teuthology.orchestra.run.smithi040:osd.7> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T18:10:25.658 INFO:tasks.cephadm:Deploying osd.8 on smithi064 with /dev/vg_nvme/lv_2... 2024-04-23T18:10:25.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:25 smithi064 ceph-mon[33062]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:25.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:25.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:25 smithi032 ceph-mon[29221]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:10:25.917 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:10:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:10:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-mon[32626]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:27.344 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:27 smithi064 ceph-mon[33062]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: from='osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:27.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:27 smithi032 ceph-mon[29221]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:10:27.867 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:10:27.459+0000 7fcf61b26700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T18:10:27.867 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:10:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:10:27.468+0000 7fcf59110700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114605 s, 915 MB/s 2024-04-23T18:10:28.087 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-23T18:10:28.323 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:28 smithi040 ceph-mon[32626]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:10:28.323 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:28 smithi040 ceph-mon[32626]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:10:28.324 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:28 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.324 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:28 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.324 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:28 smithi040 ceph-mon[32626]: from='osd.7 ' entity='osd.7' 2024-04-23T18:10:28.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:28 smithi064 ceph-mon[33062]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:10:28.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:28 smithi064 ceph-mon[33062]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:10:28.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:28 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:28 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.399 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:28 smithi064 ceph-mon[33062]: from='osd.7 ' entity='osd.7' 2024-04-23T18:10:28.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:28 smithi032 ceph-mon[29221]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:10:28.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:28 smithi032 ceph-mon[29221]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:10:28.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:28 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:28.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:28 smithi032 ceph-mon[29221]: from='osd.7 ' entity='osd.7' 2024-04-23T18:10:28.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-04-23T18:10:29.117 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:29.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:10:29.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:10:29.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] boot 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi040 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:29.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:29.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] boot 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi040 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-23T18:10:29.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:29.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:29.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: osd.7 [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] boot 2024-04-23T18:10:29.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi040 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:29.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:30.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:30 smithi040 ceph-mon[32626]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:10:30.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:30 smithi064 ceph-mon[33062]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:10:30.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:30 smithi032 ceph-mon[29221]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 -- 172.21.15.64:0/669427487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8108810 msgr2=0x7fb2b810ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 --2- 172.21.15.64:0/669427487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8108810 0x7fb2b810ac20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000b0a0 tx=0x7fb2b000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 -- 172.21.15.64:0/669427487 shutdown_connections 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 --2- 172.21.15.64:0/669427487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8108810 0x7fb2b810ac20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 --2- 172.21.15.64:0/669427487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b81035b0 0x7fb2b81059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 --2- 172.21.15.64:0/669427487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b8105ee0 0x7fb2b81082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 -- 172.21.15.64:0/669427487 >> 172.21.15.64:0/669427487 conn(0x7fb2b80fcf60 msgr2=0x7fb2b80ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 -- 172.21.15.64:0/669427487 shutdown_connections 2024-04-23T18:10:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.838+0000 7fb2be13a700 1 -- 172.21.15.64:0/669427487 wait complete. 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 Processor -- start 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 -- start start 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 0x7fb2b8102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b8108810 0x7fb2b8101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b810e550 con 0x7fb2b8108810 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b810e3d0 con 0x7fb2b81035b0 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2be13a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b810e6d0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2b6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2b6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48000/0 (socket says 172.21.15.64:48000) 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.839+0000 7fb2b6ffd700 1 -- 172.21.15.64:0/3102669771 learned_addr learned my addr 172.21.15.64:0/3102669771 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b7fff700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b8108810 0x7fb2b8101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:30.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b77fe700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 0x7fb2b8102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 -- 172.21.15.64:0/3102669771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 msgr2=0x7fb2b8102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 0x7fb2b8102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 -- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b8108810 msgr2=0x7fb2b8101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b8108810 0x7fb2b8101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 -- 172.21.15.64:0/3102669771 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b81022e0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b77fe700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 0x7fb2b8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b6ffd700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000c970 tx=0x7fb2a000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0007aa0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a0004ce0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b81a2cc0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0005510 con 0x7fb2b8105ee0 2024-04-23T18:10:30.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.840+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b81a31b0 con 0x7fb2b8105ee0 2024-04-23T18:10:30.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.841+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b8069730 con 0x7fb2b8105ee0 2024-04-23T18:10:30.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.842+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb2a0007c40 con 0x7fb2b8105ee0 2024-04-23T18:10:30.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.843+0000 7fb2b4ff9700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 0x7fb2a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:30.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.843+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7fb2a00cf860 con 0x7fb2b8105ee0 2024-04-23T18:10:30.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.843+0000 7fb2b77fe700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 0x7fb2a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:30.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.844+0000 7fb2b77fe700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 0x7fb2a40a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005950 tx=0x7fb2a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:30.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:30.847+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2a009cea0 con 0x7fb2b8105ee0 2024-04-23T18:10:31.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:31.063+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb2b8062e80 con 0x7fb2a40a3c40 2024-04-23T18:10:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:31 smithi040 ceph-mon[32626]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:31 smithi040 ceph-mon[32626]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:10:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:31 smithi064 ceph-mon[33062]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:31 smithi064 ceph-mon[33062]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:10:31.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:31 smithi032 ceph-mon[29221]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:31.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:31 smithi032 ceph-mon[29221]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:10:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:32 smithi040 ceph-mon[32626]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:10:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:32 smithi040 ceph-mon[32626]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:32.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:32.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:32.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:32 smithi064 ceph-mon[33062]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:10:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:32 smithi064 ceph-mon[33062]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:32 smithi032 ceph-mon[29221]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:10:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:32 smithi032 ceph-mon[29221]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:33 smithi040 ceph-mon[32626]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:33 smithi040 ceph-mon[32626]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:10:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:33 smithi064 ceph-mon[33062]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:33 smithi064 ceph-mon[33062]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:10:33.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:33 smithi032 ceph-mon[29221]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:33.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:33 smithi032 ceph-mon[29221]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:10:34.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:34 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/1287458909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:34 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:34 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]': finished 2024-04-23T18:10:34.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:34 smithi040 ceph-mon[32626]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:10:34.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:34 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:34.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:34 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/1287458909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:34 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:34 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]': finished 2024-04-23T18:10:34.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:34 smithi064 ceph-mon[33062]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:10:34.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:34 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:34.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:34 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/1287458909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:34 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]: dispatch 2024-04-23T18:10:34.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:34 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb2a922c-0367-4bc6-8556-983cf6b11b20"}]': finished 2024-04-23T18:10:34.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:34 smithi032 ceph-mon[29221]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:10:34.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:34 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:35.323 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:35 smithi064 ceph-mon[33062]: pgmap v186: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:35.323 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:35 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/2313366680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:35.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:35 smithi040 ceph-mon[32626]: pgmap v186: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:35.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:35 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/2313366680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:35 smithi032 ceph-mon[29221]: pgmap v186: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:35 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/2313366680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:37.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:37 smithi040 ceph-mon[32626]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:37.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:37 smithi064 ceph-mon[33062]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:37.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:37 smithi032 ceph-mon[29221]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:39.327 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:39 smithi064 ceph-mon[33062]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:39 smithi040 ceph-mon[32626]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:39.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:39 smithi032 ceph-mon[29221]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:41.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:41 smithi040 ceph-mon[32626]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:41.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:41 smithi064 ceph-mon[33062]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:41.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:41 smithi032 ceph-mon[29221]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:42.341 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:42 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:10:42.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:42 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:42 smithi064 ceph-mon[33062]: Deploying daemon osd.8 on smithi064 2024-04-23T18:10:42.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:42 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:10:42.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:42 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:42 smithi040 ceph-mon[32626]: Deploying daemon osd.8 on smithi064 2024-04-23T18:10:42.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:42 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:10:42.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:42 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:42 smithi032 ceph-mon[29221]: Deploying daemon osd.8 on smithi064 2024-04-23T18:10:43.330 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:43 smithi064 ceph-mon[33062]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:43.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:43 smithi040 ceph-mon[32626]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:43 smithi032 ceph-mon[29221]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:45.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:45 smithi064 ceph-mon[33062]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:45 smithi040 ceph-mon[32626]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:45.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:45 smithi032 ceph-mon[29221]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:47.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:47 smithi040 ceph-mon[32626]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:47.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:47 smithi064 ceph-mon[33062]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:47.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:47 smithi032 ceph-mon[29221]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:48.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:48.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:48.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:48.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:49.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:49 smithi064 ceph-mon[33062]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:49.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:49 smithi040 ceph-mon[32626]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:49.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:49 smithi032 ceph-mon[29221]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:51.480 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 8 on host 'smithi064' 2024-04-23T18:10:51.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.485+0000 7fb2b4ff9700 1 -- 172.21.15.64:0/3102669771 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb2b8062e80 con 0x7fb2a40a3c40 2024-04-23T18:10:51.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 msgr2=0x7fb2a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:51.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 0x7fb2a40a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005950 tx=0x7fb2a80058e0 comp rx=0 tx=0).stop 2024-04-23T18:10:51.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 msgr2=0x7fb2b8101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000c970 tx=0x7fb2a000cc80 comp rx=0 tx=0).stop 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 shutdown_connections 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.487+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b8108810 0x7fb2b8101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb2a40a3c40 0x7fb2a40a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b81035b0 0x7fb2b8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 --2- 172.21.15.64:0/3102669771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2b8105ee0 0x7fb2b8101560 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 >> 172.21.15.64:0/3102669771 conn(0x7fb2b80fcf60 msgr2=0x7fb2b8103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 shutdown_connections 2024-04-23T18:10:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:10:51.488+0000 7fb2be13a700 1 -- 172.21.15.64:0/3102669771 wait complete. 2024-04-23T18:10:51.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:51.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:51.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:51.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:51.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:10:52.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:52.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:52.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:52.192 DEBUG:teuthology.orchestra.run.smithi064:osd.8> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T18:10:52.195 INFO:tasks.cephadm:Deploying osd.9 on smithi032 with /dev/vg_nvme/lv_1... 2024-04-23T18:10:52.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:10:52.449 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:52.693 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:52 smithi064 ceph-mon[33062]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:10:52.694 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:52 smithi064 ceph-mon[33062]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:10:52.694 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:52 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:52.694 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:52 smithi064 ceph-mon[33062]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:52.694 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:52 smithi064 ceph-mon[33062]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:52.829 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:52 smithi032 ceph-mon[29221]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:10:52.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:52 smithi032 ceph-mon[29221]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:10:52.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:52 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:52.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:52 smithi032 ceph-mon[29221]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:52.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:52 smithi032 ceph-mon[29221]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:52.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:52 smithi040 ceph-mon[32626]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:10:52.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:52 smithi040 ceph-mon[32626]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:10:52.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:52 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:52.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:52 smithi040 ceph-mon[32626]: from='osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:52.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:52 smithi040 ceph-mon[32626]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:10:53.422 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:10:53.016+0000 7f8779ded700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T18:10:53.423 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:10:53.028+0000 7f8771bd8700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:53.768 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:53.768 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:10:53.768 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:10:53.768 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.768 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.769 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:53 smithi064 ceph-mon[33062]: from='osd.8 ' entity='osd.8' 2024-04-23T18:10:53.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:53.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:10:53.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:10:53.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:53 smithi040 ceph-mon[32626]: from='osd.8 ' entity='osd.8' 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:53.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:53 smithi032 ceph-mon[29221]: from='osd.8 ' entity='osd.8' 2024-04-23T18:10:54.689 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:10:54.689 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:10:54.690 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-23T18:10:54.690 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-23T18:10:54.690 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00891767 s, 1.2 GB/s 2024-04-23T18:10:54.693 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-23T18:10:54.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:54 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:10:54.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:54 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:10:54.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:54 smithi032 ceph-mon[29221]: osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] boot 2024-04-23T18:10:54.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:54 smithi032 ceph-mon[29221]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:10:54.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:54 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:54 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:10:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:54 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:10:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:54 smithi040 ceph-mon[32626]: osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] boot 2024-04-23T18:10:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:54 smithi040 ceph-mon[32626]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:10:54.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:54 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:54 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:10:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:54 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:10:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:54 smithi064 ceph-mon[33062]: osd.8 [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] boot 2024-04-23T18:10:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:54 smithi064 ceph-mon[33062]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:10:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:54 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:10:55.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-04-23T18:10:55.582 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:55.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:55.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:10:55.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:55.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:55.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:10:55.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:55.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:55.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:10:55.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:55.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:55.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:55.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:55.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:10:56.724 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:56 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi064 2024-04-23T18:10:56.724 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:56 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-23T18:10:56.724 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:56 smithi032 ceph-mon[29221]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:10:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:56 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi064 2024-04-23T18:10:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:56 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-23T18:10:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:56 smithi040 ceph-mon[32626]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:10:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:56 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi064 2024-04-23T18:10:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:56 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-23T18:10:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:56 smithi064 ceph-mon[33062]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 -- 172.21.15.32:0/2050313686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40102be0 msgr2=0x7fba40109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 --2- 172.21.15.32:0/2050313686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40102be0 0x7fba40109490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fba3800b0d0 tx=0x7fba3800b3e0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 -- 172.21.15.32:0/2050313686 shutdown_connections 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 --2- 172.21.15.32:0/2050313686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40102be0 0x7fba40109490 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 --2- 172.21.15.32:0/2050313686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40101040 0x7fba40101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 --2- 172.21.15.32:0/2050313686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 0x7fba401026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 -- 172.21.15.32:0/2050313686 >> 172.21.15.32:0/2050313686 conn(0x7fba400fc5e0 msgr2=0x7fba400fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 -- 172.21.15.32:0/2050313686 shutdown_connections 2024-04-23T18:10:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.280+0000 7fba45b9f700 1 -- 172.21.15.32:0/2050313686 wait complete. 2024-04-23T18:10:57.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.281+0000 7fba45b9f700 1 Processor -- start 2024-04-23T18:10:57.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.281+0000 7fba45b9f700 1 -- start start 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.281+0000 7fba45b9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.281+0000 7fba45b9f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 0x7fba40198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.281+0000 7fba45b9f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 0x7fba401a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba45b9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba4010c2c0 con 0x7fba40101040 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba45b9f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba4010c140 con 0x7fba40102240 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba45b9f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fba4010c440 con 0x7fba40102be0 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38684/0 (socket says 172.21.15.32:38684) 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 -- 172.21.15.32:0/1572906532 learned_addr learned my addr 172.21.15.32:0/1572906532 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba3ffff700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 0x7fba40198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba4539e700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 0x7fba401a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 -- 172.21.15.32:0/1572906532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 msgr2=0x7fba401a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 0x7fba401a0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 -- 172.21.15.32:0/1572906532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 msgr2=0x7fba40198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 0x7fba40198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 -- 172.21.15.32:0/1572906532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba401a06c0 con 0x7fba40101040 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba4539e700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 0x7fba401a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba44b9d700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba3400de20 tx=0x7fba3400a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3401b070 con 0x7fba40101040 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba401a09b0 con 0x7fba40101040 2024-04-23T18:10:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba3400be10 con 0x7fba40101040 2024-04-23T18:10:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.282+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340167c0 con 0x7fba40101040 2024-04-23T18:10:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.283+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba401a0ea0 con 0x7fba40101040 2024-04-23T18:10:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.284+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba40069730 con 0x7fba40101040 2024-04-23T18:10:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.285+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fba3401e070 con 0x7fba40101040 2024-04-23T18:10:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.285+0000 7fba3dffb700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 0x7fba300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.285+0000 7fba3ffff700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 0x7fba300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.285+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fba340cf900 con 0x7fba40101040 2024-04-23T18:10:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.286+0000 7fba3ffff700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 0x7fba300a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba2c005950 tx=0x7fba2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.290+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba3409cc90 con 0x7fba40101040 2024-04-23T18:10:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:10:57.507+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fba40062e80 con 0x7fba300a3c40 2024-04-23T18:10:57.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:57 smithi032 ceph-mon[29221]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:57.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:57 smithi032 ceph-mon[29221]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:10:57.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:57.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:57.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:57 smithi040 ceph-mon[32626]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:57 smithi040 ceph-mon[32626]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:10:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:57.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:57.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:57 smithi064 ceph-mon[33062]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:57.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:57 smithi064 ceph-mon[33062]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:10:57.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:57.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:57.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:58.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:58 smithi032 ceph-mon[29221]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:58.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:58 smithi032 ceph-mon[29221]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:10:58.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:58 smithi040 ceph-mon[32626]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:58.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:58 smithi040 ceph-mon[32626]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:10:58.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:58 smithi064 ceph-mon[33062]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:58.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:58 smithi064 ceph-mon[33062]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:10:59.748 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:10:59 smithi032 ceph-mon[29221]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:59.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:10:59 smithi040 ceph-mon[32626]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:10:59.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:10:59 smithi064 ceph-mon[33062]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:00.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]: dispatch 2024-04-23T18:11:00.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]': finished 2024-04-23T18:11:00.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:00 smithi040 ceph-mon[32626]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:11:00.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:00 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:00.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]: dispatch 2024-04-23T18:11:00.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]': finished 2024-04-23T18:11:00.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:00 smithi064 ceph-mon[33062]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:11:00.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:00 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:01.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]: dispatch 2024-04-23T18:11:01.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4277412458' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2db0b880-e61e-46ca-9826-421246e440fa"}]': finished 2024-04-23T18:11:01.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:00 smithi032 ceph-mon[29221]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:11:01.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:00 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:01.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:01 smithi040 ceph-mon[32626]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:11:01.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:01 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4065799263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:01.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:01 smithi064 ceph-mon[33062]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:11:01.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:01 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4065799263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:01 smithi032 ceph-mon[29221]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:11:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:01 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4065799263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:03.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:03 smithi040 ceph-mon[32626]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:03.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:03 smithi064 ceph-mon[33062]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:04.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:03 smithi032 ceph-mon[29221]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:05.792 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:05 smithi032 ceph-mon[29221]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-23T18:11:05.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:05 smithi064 ceph-mon[33062]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-23T18:11:06.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:05 smithi040 ceph-mon[32626]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-23T18:11:07.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:07 smithi032 ceph-mon[29221]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-23T18:11:07.919 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:07 smithi040 ceph-mon[32626]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-23T18:11:07.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:07 smithi064 ceph-mon[33062]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-23T18:11:08.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:11:08.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:08 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:08.977 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:11:08.978 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:08 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:09.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:11:09.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:08 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:09.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:09 smithi064 ceph-mon[33062]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:09.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:09 smithi064 ceph-mon[33062]: Deploying daemon osd.9 on smithi032 2024-04-23T18:11:09.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:09 smithi032 ceph-mon[29221]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:09.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:09 smithi032 ceph-mon[29221]: Deploying daemon osd.9 on smithi032 2024-04-23T18:11:10.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:09 smithi040 ceph-mon[32626]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:10.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:09 smithi040 ceph-mon[32626]: Deploying daemon osd.9 on smithi032 2024-04-23T18:11:11.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:11 smithi032 ceph-mon[29221]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:11.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:11 smithi064 ceph-mon[33062]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:12.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:11 smithi040 ceph-mon[32626]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:13.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:13 smithi032 ceph-mon[29221]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:13.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:13 smithi064 ceph-mon[33062]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:14.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:13 smithi040 ceph-mon[32626]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:15 smithi032 ceph-mon[29221]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:15 smithi040 ceph-mon[32626]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:15.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:15 smithi064 ceph-mon[33062]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:15.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:15.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:17 smithi032 ceph-mon[29221]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:17.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:17 smithi040 ceph-mon[32626]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:17.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:17 smithi064 ceph-mon[33062]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.374+0000 7fba3dffb700 1 -- 172.21.15.32:0/1572906532 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba40062e80 con 0x7fba300a3c40 2024-04-23T18:11:18.369 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 9 on host 'smithi032' 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 msgr2=0x7fba300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 0x7fba300a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba2c005950 tx=0x7fba2c00b410 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 msgr2=0x7fba40198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba3400de20 tx=0x7fba3400a530 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 shutdown_connections 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba40102be0 0x7fba401a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba40101040 0x7fba40198a60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fba300a3c40 0x7fba300a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 --2- 172.21.15.32:0/1572906532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba40102240 0x7fba40198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 >> 172.21.15.32:0/1572906532 conn(0x7fba400fc5e0 msgr2=0x7fba40105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 shutdown_connections 2024-04-23T18:11:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:11:18.377+0000 7fba45b9f700 1 -- 172.21.15.32:0/1572906532 wait complete. 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.686 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:18 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:11:18.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:18 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:18.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:18.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:11:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:18.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:18 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:19.016 DEBUG:teuthology.orchestra.run.smithi032:osd.9> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T18:11:19.019 INFO:tasks.cephadm:Deploying osd.10 on smithi040 with /dev/vg_nvme/lv_1... 2024-04-23T18:11:19.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:11:19.277 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:11:19.649 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:19 smithi040 ceph-mon[32626]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:19.650 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:19 smithi040 ceph-mon[32626]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:11:19.650 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:19 smithi040 ceph-mon[32626]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:11:19.650 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:19 smithi040 ceph-mon[32626]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:11:19.650 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:19 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-mon[29221]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-mon[29221]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:11:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-mon[29221]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:11:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-mon[29221]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:11:19.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:19.759 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:11:19.524+0000 7fb1fed4a700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T18:11:19.759 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:11:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:11:19.538+0000 7fb1fab3d700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:19 smithi064 ceph-mon[33062]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:19 smithi064 ceph-mon[33062]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:11:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:19 smithi064 ceph-mon[33062]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:11:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:19 smithi064 ceph-mon[33062]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-23T18:11:19.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:19 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.725 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:20 smithi040 ceph-mon[32626]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:11:20.725 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:20 smithi040 ceph-mon[32626]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:11:20.725 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.725 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:20 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:20 smithi032 ceph-mon[29221]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:11:20.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:20 smithi032 ceph-mon[29221]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:11:20.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.816 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:20 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:20 smithi064 ceph-mon[33062]: from='osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-23T18:11:20.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:20 smithi064 ceph-mon[33062]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:11:20.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:20.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:20 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128123 s, 818 MB/s 2024-04-23T18:11:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] boot 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:11:21.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:21 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] boot 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:11:21.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:21 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:11:21.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:11:21.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:11:21.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:21.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: osd.9 [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] boot 2024-04-23T18:11:21.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:11:21.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:21 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:11:22.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-04-23T18:11:22.381 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.652 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:22.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:22.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:22 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:11:22.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:22.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:22 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:23.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:23.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:23.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:22 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:23.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:23 smithi032 ceph-mon[29221]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:23.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:23 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi032 2024-04-23T18:11:23.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:23 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-23T18:11:23.827 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:23 smithi040 ceph-mon[32626]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:23.827 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:23 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi032 2024-04-23T18:11:23.827 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:23 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-23T18:11:23.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:23 smithi064 ceph-mon[33062]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:23.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:23 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi032 2024-04-23T18:11:23.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:23 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-23T18:11:24.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 -- 172.21.15.40:0/2393706328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 msgr2=0x7f7e7c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 --2- 172.21.15.40:0/2393706328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c1082b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70009ca0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 -- 172.21.15.40:0/2393706328 shutdown_connections 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 --2- 172.21.15.40:0/2393706328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e7c1087f0 0x7f7e7c10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 --2- 172.21.15.40:0/2393706328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 --2- 172.21.15.40:0/2393706328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c1082b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 -- 172.21.15.40:0/2393706328 >> 172.21.15.40:0/2393706328 conn(0x7f7e7c0fcf40 msgr2=0x7f7e7c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 -- 172.21.15.40:0/2393706328 shutdown_connections 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.195+0000 7f7e80eff700 1 -- 172.21.15.40:0/2393706328 wait complete. 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 Processor -- start 2024-04-23T18:11:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 -- start start 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c19aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e7c1087f0 0x7f7e7c1a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c10e5b0 con 0x7f7e7c103590 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c10e430 con 0x7f7e7c105ec0 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.196+0000 7f7e80eff700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c10e730 con 0x7f7e7c1087f0 2024-04-23T18:11:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47990/0 (socket says 172.21.15.40:47990) 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 -- 172.21.15.40:0/303795970 learned_addr learned my addr 172.21.15.40:0/303795970 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e7a59c700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c19aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 -- 172.21.15.40:0/303795970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e7c1087f0 msgr2=0x7f7e7c1a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e7c1087f0 0x7f7e7c1a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 -- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 msgr2=0x7f7e7c19aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c19aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 -- 172.21.15.40:0/303795970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c1a2b00 con 0x7f7e7c105ec0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e7a59c700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c19aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e79d9b700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e700036f0 tx=0x7f7e700039f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70027070 con 0x7f7e7c105ec0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c1a2d90 con 0x7f7e7c105ec0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e70022470 con 0x7f7e7c105ec0 2024-04-23T18:11:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000f6c0 con 0x7f7e7c105ec0 2024-04-23T18:11:24.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.197+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c1a3260 con 0x7f7e7c105ec0 2024-04-23T18:11:24.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.198+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c069730 con 0x7f7e7c105ec0 2024-04-23T18:11:24.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.199+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7e7002a070 con 0x7f7e7c105ec0 2024-04-23T18:11:24.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.200+0000 7f7e6b7fe700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 0x7f7e640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.200+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f7e700d0f30 con 0x7f7e7c105ec0 2024-04-23T18:11:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.200+0000 7f7e7a59c700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 0x7f7e640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.201+0000 7f7e7a59c700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 0x7f7e640a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0099d0 tx=0x7f7e6c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.204+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e70013070 con 0x7f7e7c105ec0 2024-04-23T18:11:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:24.424+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7e7c062e80 con 0x7f7e640a3c40 2024-04-23T18:11:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:24.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:24.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:24.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:25.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:25.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:25.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:25.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:25 smithi040 ceph-mon[32626]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:25.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:25 smithi040 ceph-mon[32626]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:25.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:25 smithi064 ceph-mon[33062]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:25.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:25 smithi064 ceph-mon[33062]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:25 smithi032 ceph-mon[29221]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:26.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:25 smithi032 ceph-mon[29221]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/3086537855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]': finished 2024-04-23T18:11:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:11:27.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:27 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:27.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:27.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/3086537855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:27.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:27.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]': finished 2024-04-23T18:11:27.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:11:27.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:27 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/3086537855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]: dispatch 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2116ca5-2fe8-405c-bfeb-0eac6b746fbc"}]': finished 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:11:28.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:27 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:28.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:28 smithi040 ceph-mon[32626]: from='client.? 172.21.15.40:0/2943861031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:28.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:28 smithi064 ceph-mon[33062]: from='client.? 172.21.15.40:0/2943861031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:29.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:28 smithi032 ceph-mon[29221]: from='client.? 172.21.15.40:0/2943861031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:29.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:29 smithi040 ceph-mon[32626]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:29.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:29 smithi064 ceph-mon[33062]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:30.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:29 smithi032 ceph-mon[29221]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:31.847 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:31 smithi040 ceph-mon[32626]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:31.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:31 smithi064 ceph-mon[33062]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:32.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:31 smithi032 ceph-mon[29221]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:33.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:33 smithi040 ceph-mon[32626]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:33.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:33 smithi064 ceph-mon[33062]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:34.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:33 smithi032 ceph-mon[29221]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:35.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:35 smithi040 ceph-mon[32626]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:35.731 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:11:35.731 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:35.731 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:35 smithi040 ceph-mon[32626]: Deploying daemon osd.10 on smithi040 2024-04-23T18:11:35.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:35 smithi064 ceph-mon[33062]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:35.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:11:35.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:35.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:35 smithi064 ceph-mon[33062]: Deploying daemon osd.10 on smithi040 2024-04-23T18:11:36.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:35 smithi032 ceph-mon[29221]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:36.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:11:36.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:36.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:35 smithi032 ceph-mon[29221]: Deploying daemon osd.10 on smithi040 2024-04-23T18:11:37.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:37 smithi040 ceph-mon[32626]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:37.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:37 smithi064 ceph-mon[33062]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:38.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:37 smithi032 ceph-mon[29221]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:39.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:39 smithi040 ceph-mon[32626]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:39.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:39 smithi064 ceph-mon[33062]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:40.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:39 smithi032 ceph-mon[29221]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:42.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:41 smithi040 ceph-mon[32626]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:42.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:41 smithi064 ceph-mon[33062]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:41 smithi032 ceph-mon[29221]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:43.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:43 smithi040 ceph-mon[32626]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:44.082 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:43 smithi032 ceph-mon[29221]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:44.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:43 smithi064 ceph-mon[33062]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:44.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.694+0000 7f7e6b7fe700 1 -- 172.21.15.40:0/303795970 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7e7c062e80 con 0x7f7e640a3c40 2024-04-23T18:11:44.690 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 10 on host 'smithi040' 2024-04-23T18:11:44.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.696+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 msgr2=0x7f7e640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:44.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.696+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 0x7f7e640a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0099d0 tx=0x7f7e6c008040 comp rx=0 tx=0).stop 2024-04-23T18:11:44.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.696+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 msgr2=0x7f7e7c19b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.696+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e700036f0 tx=0x7f7e700039f0 comp rx=0 tx=0).stop 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 shutdown_connections 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e7c1087f0 0x7f7e7c1a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e7c103590 0x7f7e7c19aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7e640a3c40 0x7f7e640a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 --2- 172.21.15.40:0/303795970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7c105ec0 0x7f7e7c19b3e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 >> 172.21.15.40:0/303795970 conn(0x7f7e7c0fcf40 msgr2=0x7f7e7c102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 shutdown_connections 2024-04-23T18:11:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-23T18:11:44.697+0000 7f7e80eff700 1 -- 172.21.15.40:0/303795970 wait complete. 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:45.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:45.417 DEBUG:teuthology.orchestra.run.smithi040:osd.10> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T18:11:45.421 INFO:tasks.cephadm:Deploying osd.11 on smithi064 with /dev/vg_nvme/lv_1... 2024-04-23T18:11:45.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:11:45.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:11:45.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:45.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:45.680 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:46.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:46 smithi064 ceph-mon[33062]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:11:46.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:46 smithi064 ceph-mon[33062]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:46.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:46 smithi064 ceph-mon[33062]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:11:46.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:46 smithi064 ceph-mon[33062]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-mon[32626]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-mon[32626]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-mon[32626]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:46.923 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-mon[32626]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:46.924 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:11:46.668+0000 7fb7b4d54700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T18:11:46.924 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:11:46 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:11:46.685+0000 7fb7b0b47700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:46 smithi032 ceph-mon[29221]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:11:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:46 smithi032 ceph-mon[29221]: from='osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:46 smithi032 ceph-mon[29221]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:11:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:46 smithi032 ceph-mon[29221]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-23T18:11:47.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:11:47.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:11:47.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:47.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:11:47.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:11:47.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:47.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:47.974 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:11:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:11:47.975 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-23T18:11:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-23T18:11:47.975 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130787 s, 802 MB/s 2024-04-23T18:11:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:48.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:11:48.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:48.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:48.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-04-23T18:11:48.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:48.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:48 smithi064 ceph-mon[33062]: osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] boot 2024-04-23T18:11:48.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:48 smithi064 ceph-mon[33062]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:11:48.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:48 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:49.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:49.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:48 smithi032 ceph-mon[29221]: osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] boot 2024-04-23T18:11:49.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:48 smithi032 ceph-mon[29221]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:11:49.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:48 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:49.049 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:49.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:49.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:48 smithi040 ceph-mon[32626]: osd.10 [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] boot 2024-04-23T18:11:49.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:48 smithi040 ceph-mon[32626]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:11:49.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:48 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:11:49.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:49.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi040 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:49.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi040 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-23T18:11:50.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:50.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:50.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:11:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:11:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi040 2024-04-23T18:11:50.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:50.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:11:50.949 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:50 smithi064 ceph-mon[33062]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:11:50.949 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:50 smithi064 ceph-mon[33062]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:11:51.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:50 smithi032 ceph-mon[29221]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:11:51.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:50 smithi032 ceph-mon[29221]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 -- 172.21.15.64:0/1919296294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 msgr2=0x7f0634102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 --2- 172.21.15.64:0/1919296294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f0634102ee0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0624009990 tx=0x7f0624009ca0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 -- 172.21.15.64:0/1919296294 shutdown_connections 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 --2- 172.21.15.64:0/1919296294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634104660 0x7f063410af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 --2- 172.21.15.64:0/1919296294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f0634102ee0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 --2- 172.21.15.64:0/1919296294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103cc0 0x7f0634104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 -- 172.21.15.64:0/1919296294 >> 172.21.15.64:0/1919296294 conn(0x7f06340fe060 msgr2=0x7f06341004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 -- 172.21.15.64:0/1919296294 shutdown_connections 2024-04-23T18:11:51.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.066+0000 7f0639352700 1 -- 172.21.15.64:0/1919296294 wait complete. 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.067+0000 7f0639352700 1 Processor -- start 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.067+0000 7f0639352700 1 -- start start 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103cc0 0x7f0634074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 0x7f0634075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f063410e530 con 0x7f0634104660 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f063410e3b0 con 0x7f0634103cc0 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f063410e6b0 con 0x7f0634102ac0 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43422/0 (socket says 172.21.15.64:43422) 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 -- 172.21.15.64:0/127882653 learned_addr learned my addr 172.21.15.64:0/127882653 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:11:51.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f06327fc700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103cc0 0x7f0634074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f06337fe700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 0x7f0634075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 -- 172.21.15.64:0/127882653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103cc0 msgr2=0x7f0634074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103cc0 0x7f0634074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 -- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 msgr2=0x7f0634075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 0x7f0634075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 -- 172.21.15.64:0/127882653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0634075c20 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f06337fe700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 0x7f0634075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0632ffd700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f0624009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624027070 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06240041c0 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06341a2c70 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.068+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062400f460 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.069+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341a30c0 con 0x7f0634102ac0 2024-04-23T18:11:51.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.069+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0634069730 con 0x7f0634102ac0 2024-04-23T18:11:51.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.071+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f062402a070 con 0x7f0634102ac0 2024-04-23T18:11:51.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.071+0000 7f061bfff700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 0x7f061c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:51.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.071+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f06240d1490 con 0x7f0634102ac0 2024-04-23T18:11:51.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.072+0000 7f06327fc700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 0x7f061c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:51.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.072+0000 7f06327fc700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 0x7f061c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0628005ea0 tx=0x7f0628005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:51.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.075+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0624031070 con 0x7f0634102ac0 2024-04-23T18:11:51.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:50 smithi040 ceph-mon[32626]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:11:51.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:50 smithi040 ceph-mon[32626]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:11:51.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:11:51.293+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0634062e80 con 0x7f061c0a3c40 2024-04-23T18:11:52.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:51 smithi032 ceph-mon[29221]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:52.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:51 smithi032 ceph-mon[29221]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:11:52.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:51 smithi064 ceph-mon[33062]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:52.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:52.034 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:52.034 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:52.034 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:51 smithi064 ceph-mon[33062]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:11:52.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:51 smithi040 ceph-mon[32626]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:52.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:52.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:52.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:52.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:51 smithi040 ceph-mon[32626]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:11:52.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:52 smithi064 ceph-mon[33062]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:52 smithi032 ceph-mon[29221]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:53.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:52 smithi040 ceph-mon[32626]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:54.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:53 smithi032 ceph-mon[29221]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:54.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:53 smithi040 ceph-mon[32626]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:54.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:53 smithi064 ceph-mon[33062]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:54 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/564537836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:54 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:54 smithi064 ceph-mon[33062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]': finished 2024-04-23T18:11:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:54 smithi064 ceph-mon[33062]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:11:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:54 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:11:55.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:54 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/564537836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:55.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:54 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:55.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:54 smithi032 ceph-mon[29221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]': finished 2024-04-23T18:11:55.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:54 smithi032 ceph-mon[29221]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:11:55.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:54 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:11:55.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:54 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/564537836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:55.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:54 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]: dispatch 2024-04-23T18:11:55.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:54 smithi040 ceph-mon[32626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d5a0032-0c95-4c63-b06a-340f80f978de"}]': finished 2024-04-23T18:11:55.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:54 smithi040 ceph-mon[32626]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:11:55.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:54 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:11:56.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:55 smithi032 ceph-mon[29221]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:56.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:55 smithi032 ceph-mon[29221]: from='client.? 172.21.15.64:0/2101053459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:56.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:55 smithi040 ceph-mon[32626]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:56.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:55 smithi040 ceph-mon[32626]: from='client.? 172.21.15.64:0/2101053459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:55 smithi064 ceph-mon[33062]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:11:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:55 smithi064 ceph-mon[33062]: from='client.? 172.21.15.64:0/2101053459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:58.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:57 smithi032 ceph-mon[29221]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:58.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:57 smithi040 ceph-mon[32626]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:58.168 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:57 smithi064 ceph-mon[33062]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:59.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:11:59 smithi064 ceph-mon[33062]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:12:00.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:11:59 smithi032 ceph-mon[29221]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:12:00.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:11:59 smithi040 ceph-mon[32626]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:12:01.851 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:01 smithi064 ceph-mon[33062]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:12:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:01 smithi032 ceph-mon[29221]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:12:02.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:01 smithi040 ceph-mon[32626]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:12:02.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:02 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:12:02.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:02 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.796 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:02 smithi064 ceph-mon[33062]: Deploying daemon osd.11 on smithi064 2024-04-23T18:12:03.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:02 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:12:03.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:02 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:03.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:02 smithi032 ceph-mon[29221]: Deploying daemon osd.11 on smithi064 2024-04-23T18:12:03.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:02 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:12:03.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:02 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:03.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:02 smithi040 ceph-mon[32626]: Deploying daemon osd.11 on smithi064 2024-04-23T18:12:03.895 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:03 smithi064 ceph-mon[33062]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:04.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:03 smithi032 ceph-mon[29221]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:04.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:03 smithi040 ceph-mon[32626]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:05.917 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:05 smithi064 ceph-mon[33062]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:06.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:05 smithi032 ceph-mon[29221]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:06.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:05 smithi040 ceph-mon[32626]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:12:07.897 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:07 smithi064 ceph-mon[33062]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:08.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:07 smithi032 ceph-mon[29221]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:08.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:07 smithi040 ceph-mon[32626]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:09 smithi032 ceph-mon[29221]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:09 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:09.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:09 smithi064 ceph-mon[33062]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:09.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:09 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:09 smithi040 ceph-mon[32626]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:09.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:09 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:11 smithi064 ceph-mon[33062]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:11.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:11 smithi032 ceph-mon[29221]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:11.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:11 smithi040 ceph-mon[32626]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:12.215 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 11 on host 'smithi064' 2024-04-23T18:12:12.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.220+0000 7f061bfff700 1 -- 172.21.15.64:0/127882653 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0634062e80 con 0x7f061c0a3c40 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 msgr2=0x7f061c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 0x7f061c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0628005ea0 tx=0x7f0628005e30 comp rx=0 tx=0).stop 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 msgr2=0x7f06340767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f0624009580 comp rx=0 tx=0).stop 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 shutdown_connections 2024-04-23T18:12:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634104660 0x7f0634075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634102ac0 0x7f06340767e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3c40 0x7f061c0a6100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 --2- 172.21.15.64:0/127882653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103cc0 0x7f0634074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 >> 172.21.15.64:0/127882653 conn(0x7f06340fe060 msgr2=0x7f0634106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 shutdown_connections 2024-04-23T18:12:12.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:12.223+0000 7f0639352700 1 -- 172.21.15.64:0/127882653 wait complete. 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:12 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:12 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:12.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:12 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:12.894 DEBUG:teuthology.orchestra.run.smithi064:osd.11> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T18:12:12.897 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T18:12:12.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T18:12:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:12:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:12:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:12:13.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:13.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:13.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-mon[33062]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:13.665 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:13.665 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:12:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:12:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:13.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:13 smithi032 ceph-mon[29221]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: from='osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:13 smithi040 ceph-mon[32626]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-23T18:12:14.150 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:12:13.794+0000 7fce37d3d700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T18:12:14.150 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:12:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:12:13.808+0000 7fce2fb28700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:14 smithi064 ceph-mon[33062]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:12:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:14 smithi064 ceph-mon[33062]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:12:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:14 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:14 smithi064 ceph-mon[33062]: from='osd.11 ' entity='osd.11' 2024-04-23T18:12:14.706 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:14 smithi032 ceph-mon[29221]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:12:14.706 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:14 smithi032 ceph-mon[29221]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:12:14.706 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.706 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:14 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.706 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:14 smithi032 ceph-mon[29221]: from='osd.11 ' entity='osd.11' 2024-04-23T18:12:14.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:14 smithi040 ceph-mon[32626]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-23T18:12:14.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:14 smithi040 ceph-mon[32626]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:12:14.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:14 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:14.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:14 smithi040 ceph-mon[32626]: from='osd.11 ' entity='osd.11' 2024-04-23T18:12:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- 172.21.15.32:0/1220723229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0106680 msgr2=0x7fc1b0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 --2- 172.21.15.32:0/1220723229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0106680 0x7fc1b0106b00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800b0a0 tx=0x7fc1a800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- 172.21.15.32:0/1220723229 shutdown_connections 2024-04-23T18:12:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 --2- 172.21.15.32:0/1220723229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0106680 0x7fc1b0106b00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 --2- 172.21.15.32:0/1220723229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 0x7fc1b0103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 --2- 172.21.15.32:0/1220723229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0103cb0 0x7fc1b0106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- 172.21.15.32:0/1220723229 >> 172.21.15.32:0/1220723229 conn(0x7fc1b00fcf60 msgr2=0x7fc1b00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- 172.21.15.32:0/1220723229 shutdown_connections 2024-04-23T18:12:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- 172.21.15.32:0/1220723229 wait complete. 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 Processor -- start 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.070+0000 7fc1b8626700 1 -- start start 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 0x7fc1b019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0106680 0x7fc1b01a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b0114ab0 con 0x7fc1b0103cb0 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b0114930 con 0x7fc1b0106680 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b0114c30 con 0x7fc1b0101340 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38416/0 (socket says 172.21.15.32:38416) 2024-04-23T18:12:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 -- 172.21.15.32:0/2107711658 learned_addr learned my addr 172.21.15.32:0/2107711658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b63c2700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 0x7fc1b019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b6bc3700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0106680 0x7fc1b01a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 -- 172.21.15.32:0/2107711658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 msgr2=0x7fc1b019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 0x7fc1b019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 -- 172.21.15.32:0/2107711658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0106680 msgr2=0x7fc1b01a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0106680 0x7fc1b01a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 -- 172.21.15.32:0/2107711658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b01a6d20 con 0x7fc1b0103cb0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b5bc1700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9d0 tx=0x7fc1a000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000cbd0 con 0x7fc1b0103cb0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1a000ee70 con 0x7fc1b0103cb0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b01a6fb0 con 0x7fc1b0103cb0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.071+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0018680 con 0x7fc1b0103cb0 2024-04-23T18:12:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.072+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b01a74c0 con 0x7fc1b0103cb0 2024-04-23T18:12:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.073+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b0069730 con 0x7fc1b0103cb0 2024-04-23T18:12:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.074+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc1a0010430 con 0x7fc1b0103cb0 2024-04-23T18:12:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.075+0000 7fc1a77fe700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 0x7fc19c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.075+0000 7fc1b63c2700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 0x7fc19c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.075+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc1a00d03b0 con 0x7fc1b0103cb0 2024-04-23T18:12:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.076+0000 7fc1b63c2700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 0x7fc19c0a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac007c60 tx=0x7fc1ac0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:15.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.079+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1a009d230 con 0x7fc1b0103cb0 2024-04-23T18:12:15.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.291+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc1b002e190 con 0x7fc1b0103cb0 2024-04-23T18:12:15.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.291+0000 7fc1a77fe700 1 -- 172.21.15.32:0/2107711658 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fc1a009cb30 con 0x7fc1b0103cb0 2024-04-23T18:12:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:15.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.293+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 msgr2=0x7fc19c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:15.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.293+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 0x7fc19c0a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac007c60 tx=0x7fc1ac0073d0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.293+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 msgr2=0x7fc1b019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.293+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9d0 tx=0x7fc1a000ece0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 shutdown_connections 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc19c0a3c40 0x7fc19c0a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b0106680 0x7fc1b01a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1b0101340 0x7fc1b019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 --2- 172.21.15.32:0/2107711658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b0103cb0 0x7fc1b019f600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 >> 172.21.15.32:0/2107711658 conn(0x7fc1b00fcf60 msgr2=0x7fc1b0101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 shutdown_connections 2024-04-23T18:12:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:15.294+0000 7fc1b8626700 1 -- 172.21.15.32:0/2107711658 wait complete. 2024-04-23T18:12:15.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: purged_snaps scrub starts 2024-04-23T18:12:15.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: purged_snaps scrub ok 2024-04-23T18:12:15.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:15.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] boot 2024-04-23T18:12:15.584 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:12:15.584 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.584 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:15 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2107711658' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:12:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: purged_snaps scrub starts 2024-04-23T18:12:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: purged_snaps scrub ok 2024-04-23T18:12:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] boot 2024-04-23T18:12:15.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:12:15.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:15 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2107711658' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: purged_snaps scrub starts 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: purged_snaps scrub ok 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: osd.11 [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] boot 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:12:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:12:15.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:15 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2107711658' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:12:16.104 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713895934,"num_in_osds":12,"osd_in_since":1713895914,"num_remapped_pgs":0} 2024-04-23T18:12:16.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:12:16.363 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:12:17.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:12:17.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:17.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:16 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:12:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:17.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:16 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:17.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:17.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:17.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:16 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:17.912 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:17 smithi032 ceph-mon[29221]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:17.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:17 smithi032 ceph-mon[29221]: Detected new or changed devices on smithi064 2024-04-23T18:12:17.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:17 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-23T18:12:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:17 smithi040 ceph-mon[32626]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:17 smithi040 ceph-mon[32626]: Detected new or changed devices on smithi064 2024-04-23T18:12:18.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:17 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-23T18:12:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:17 smithi064 ceph-mon[33062]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:17 smithi064 ceph-mon[33062]: Detected new or changed devices on smithi064 2024-04-23T18:12:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:17 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 -- 172.21.15.32:0/1978436321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006b0f0 msgr2=0x7fc84010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 --2- 172.21.15.32:0/1978436321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006b0f0 0x7fc84010f900 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc83800b0a0 tx=0x7fc83800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 -- 172.21.15.32:0/1978436321 shutdown_connections 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 --2- 172.21.15.32:0/1978436321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006b0f0 0x7fc84010f900 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 --2- 172.21.15.32:0/1978436321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc840108a30 0x7fc840108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 --2- 172.21.15.32:0/1978436321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc84006a750 0x7fc84006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.295+0000 7fc844a42700 1 -- 172.21.15.32:0/1978436321 >> 172.21.15.32:0/1978436321 conn(0x7fc8400788f0 msgr2=0x7fc840078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.296+0000 7fc844a42700 1 -- 172.21.15.32:0/1978436321 shutdown_connections 2024-04-23T18:12:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.296+0000 7fc844a42700 1 -- 172.21.15.32:0/1978436321 wait complete. 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 Processor -- start 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 -- start start 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84006b0f0 0x7fc840199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 0x7fc84019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc840112bd0 con 0x7fc84006a750 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc840112a50 con 0x7fc84006b0f0 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc844a42700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc840112d50 con 0x7fc840108a30 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54360/0 (socket says 172.21.15.32:54360) 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 -- 172.21.15.32:0/987633401 learned_addr learned my addr 172.21.15.32:0/987633401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83dd9b700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84006b0f0 0x7fc840199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83ed9d700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 0x7fc84019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 -- 172.21.15.32:0/987633401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 msgr2=0x7fc84019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 0x7fc84019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 -- 172.21.15.32:0/987633401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84006b0f0 msgr2=0x7fc840199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84006b0f0 0x7fc840199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83e59c700 1 -- 172.21.15.32:0/987633401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84019e0d0 con 0x7fc84006a750 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.297+0000 7fc83ed9d700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 0x7fc84019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc83e59c700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e9a0 tx=0x7fc83000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83000cbd0 con 0x7fc84006a750 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc83000cd70 con 0x7fc84006a750 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830018680 con 0x7fc84006a750 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84019e3c0 con 0x7fc84006a750 2024-04-23T18:12:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.298+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8401a5c40 con 0x7fc84006a750 2024-04-23T18:12:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.299+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc840069730 con 0x7fc84006a750 2024-04-23T18:12:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.300+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc830010430 con 0x7fc84006a750 2024-04-23T18:12:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.301+0000 7fc82f7fe700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 0x7fc8280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.301+0000 7fc83dd9b700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 0x7fc8280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.301+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc8300d05f0 con 0x7fc84006a750 2024-04-23T18:12:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.301+0000 7fc83dd9b700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 0x7fc8280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8401084f0 tx=0x7fc83400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:18.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.305+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc83009d3d0 con 0x7fc84006a750 2024-04-23T18:12:18.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.514+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc84002df80 con 0x7fc84006a750 2024-04-23T18:12:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.515+0000 7fc82f7fe700 1 -- 172.21.15.32:0/987633401 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18385 (secure 0 0 0) 0x7fc83009ccd0 con 0x7fc84006a750 2024-04-23T18:12:18.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:18.510 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:12:15.803236+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 msgr2=0x7fc8280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 0x7fc8280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8401084f0 tx=0x7fc83400a300 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 msgr2=0x7fc8401990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e9a0 tx=0x7fc83000ecb0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 shutdown_connections 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc840108a30 0x7fc84019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc84006a750 0x7fc8401990c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc8280a3cc0 0x7fc8280a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 --2- 172.21.15.32:0/987633401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84006b0f0 0x7fc840199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 >> 172.21.15.32:0/987633401 conn(0x7fc8400788f0 msgr2=0x7fc8400ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 shutdown_connections 2024-04-23T18:12:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:18.517+0000 7fc844a42700 1 -- 172.21.15.32:0/987633401 wait complete. 2024-04-23T18:12:19.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:18 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/987633401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:19.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:18 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/987633401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:19.142 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:12:19.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T18:12:19.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:18 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/987633401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:19.388 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:12:20.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:19 smithi040 ceph-mon[32626]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:20.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:19 smithi064 ceph-mon[33062]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:20.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:19 smithi032 ceph-mon[29221]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 -- 172.21.15.32:0/87645276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4107960 msgr2=0x7f77e4107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 --2- 172.21.15.32:0/87645276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4107960 0x7f77e4107d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f77d4009990 tx=0x7f77d4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 -- 172.21.15.32:0/87645276 shutdown_connections 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 --2- 172.21.15.32:0/87645276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4102320 0x7f77e410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 --2- 172.21.15.32:0/87645276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4107960 0x7f77e4107d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 --2- 172.21.15.32:0/87645276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4101960 0x7f77e4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 -- 172.21.15.32:0/87645276 >> 172.21.15.32:0/87645276 conn(0x7f77e40789e0 msgr2=0x7f77e4078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 -- 172.21.15.32:0/87645276 shutdown_connections 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 -- 172.21.15.32:0/87645276 wait complete. 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.140+0000 7f77ea27c700 1 Processor -- start 2024-04-23T18:12:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 -- start start 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4102320 0x7f77e419b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4107960 0x7f77e419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77e4111b40 con 0x7f77e4101960 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77e41119c0 con 0x7f77e4102320 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77ea27c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f77e4111cc0 con 0x7f77e4107960 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54382/0 (socket says 172.21.15.32:54382) 2024-04-23T18:12:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 -- 172.21.15.32:0/928524917 learned_addr learned my addr 172.21.15.32:0/928524917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e8819700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4107960 0x7f77e419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e37fe700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4102320 0x7f77e419b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 -- 172.21.15.32:0/928524917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4107960 msgr2=0x7f77e419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4107960 0x7f77e419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 -- 172.21.15.32:0/928524917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4102320 msgr2=0x7f77e419b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4102320 0x7f77e419b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.141+0000 7f77e3fff700 1 -- 172.21.15.32:0/928524917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e41a0390 con 0x7f77e4101960 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77e3fff700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000c00 tx=0x7f77d40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4027070 con 0x7f77e4101960 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e41a0620 con 0x7f77e4101960 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d4003f20 con 0x7f77e4101960 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4022ca0 con 0x7f77e4101960 2024-04-23T18:12:21.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.142+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e41a7f00 con 0x7f77e4101960 2024-04-23T18:12:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.143+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e4069730 con 0x7f77e4101960 2024-04-23T18:12:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.144+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f77d402a070 con 0x7f77e4101960 2024-04-23T18:12:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.144+0000 7f77e17fa700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 0x7f77cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.145+0000 7f77e37fe700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 0x7f77cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:21.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.145+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f77d40d0e60 con 0x7f77e4101960 2024-04-23T18:12:21.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.145+0000 7f77e37fe700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 0x7f77cc0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77e4079370 tx=0x7f77d800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.149+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77d405eff0 con 0x7f77e4101960 2024-04-23T18:12:21.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.355+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77e402df80 con 0x7f77e4101960 2024-04-23T18:12:21.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.355+0000 7f77e17fa700 1 -- 172.21.15.32:0/928524917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f77d4013070 con 0x7f77e4101960 2024-04-23T18:12:21.349 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 msgr2=0x7f77cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 0x7f77cc0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77e4079370 tx=0x7f77d800a300 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 msgr2=0x7f77e419b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000c00 tx=0x7f77d40047b0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 shutdown_connections 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e4107960 0x7f77e419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77e4101960 0x7f77e419b380 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f77cc0a3c40 0x7f77cc0a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 --2- 172.21.15.32:0/928524917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77e4102320 0x7f77e419b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 >> 172.21.15.32:0/928524917 conn(0x7f77e40789e0 msgr2=0x7f77e410cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 shutdown_connections 2024-04-23T18:12:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:21.357+0000 7f77ea27c700 1 -- 172.21.15.32:0/928524917 wait complete. 2024-04-23T18:12:22.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:21 smithi040 ceph-mon[32626]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:22.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:21 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/928524917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:22.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:21 smithi032 ceph-mon[29221]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:22.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:21 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/928524917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:22.169 INFO:tasks.cephadm:Adding mds.a on smithi032 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.d on smithi032 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.g on smithi032 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.j on smithi032 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.b on smithi040 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.e on smithi040 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.h on smithi040 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.k on smithi040 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.c on smithi064 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.f on smithi064 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.i on smithi064 2024-04-23T18:12:22.170 INFO:tasks.cephadm:Adding mds.l on smithi064 2024-04-23T18:12:22.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l' 2024-04-23T18:12:22.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:21 smithi064 ceph-mon[33062]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:22.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:21 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/928524917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:22.430 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:12:23.896 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:23 smithi064 ceph-mon[33062]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:24.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:23 smithi040 ceph-mon[32626]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 -- 172.21.15.64:0/3472615361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0102660 msgr2=0x7f2ee0102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/3472615361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0102660 0x7f2ee0102ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800b0a0 tx=0x7f2ed800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 -- 172.21.15.64:0/3472615361 shutdown_connections 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/3472615361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0102660 0x7f2ee0102ae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/3472615361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0101340 0x7f2ee0101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/3472615361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101ca0 0x7f2ee0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 -- 172.21.15.64:0/3472615361 >> 172.21.15.64:0/3472615361 conn(0x7f2ee00fcf60 msgr2=0x7f2ee00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 -- 172.21.15.64:0/3472615361 shutdown_connections 2024-04-23T18:12:24.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.181+0000 7f2ee51ea700 1 -- 172.21.15.64:0/3472615361 wait complete. 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.182+0000 7f2ee51ea700 1 Processor -- start 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.182+0000 7f2ee51ea700 1 -- start start 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101340 0x7f2ee019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 0x7f2ee01a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0110740 con 0x7f2ee0101340 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee01105c0 con 0x7f2ee0102660 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ee51ea700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee01108c0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59470/0 (socket says 172.21.15.64:59470) 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 -- 172.21.15.64:0/1423747398 learned_addr learned my addr 172.21.15.64:0/1423747398 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2edf59e700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 0x7f2ee01a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:24.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2eded9d700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101340 0x7f2ee019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 -- 172.21.15.64:0/1423747398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 msgr2=0x7f2ee01a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 0x7f2ee01a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 -- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101340 msgr2=0x7f2ee019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101340 0x7f2ee019ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 -- 172.21.15.64:0/1423747398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee01a29c0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2edf59e700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 0x7f2ee01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.183+0000 7f2ede59c700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e9d0 tx=0x7f2ed000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.184+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000cbd0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.184+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ed000ee70 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.184+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee01a2cb0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.184+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0018680 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.184+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee01a31a0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.185+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee0069730 con 0x7f2ee0101ca0 2024-04-23T18:12:24.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.186+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2ed0010430 con 0x7f2ee0101ca0 2024-04-23T18:12:24.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.187+0000 7f2ec7fff700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 0x7f2ec80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:24.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.187+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2ed00d0440 con 0x7f2ee0101ca0 2024-04-23T18:12:24.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.187+0000 7f2eded9d700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 0x7f2ec80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:24.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.187+0000 7f2eded9d700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 0x7f2ec80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4009c60 tx=0x7f2ed4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:24.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.190+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ed009d2c0 con 0x7f2ee0101ca0 2024-04-23T18:12:24.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:23 smithi032 ceph-mon[29221]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:24.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.411+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l", "target": ["mon-mgr", ""]}) -- 0x7f2ee0063140 con 0x7f2ec80a3c40 2024-04-23T18:12:24.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.419+0000 7f2ec7fff700 1 -- 172.21.15.64:0/1423747398 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f2ee0063140 con 0x7f2ec80a3c40 2024-04-23T18:12:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled mds.all update... 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 msgr2=0x7f2ec80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 0x7f2ec80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4009c60 tx=0x7f2ed4009260 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 msgr2=0x7f2ee019b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e9d0 tx=0x7f2ed000ece0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 shutdown_connections 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ec80a3c40 0x7f2ec80a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee0102660 0x7f2ee01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0101340 0x7f2ee019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 --2- 172.21.15.64:0/1423747398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ee0101ca0 0x7f2ee019b2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 >> 172.21.15.64:0/1423747398 conn(0x7f2ee00fcf60 msgr2=0x7f2ee00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:24.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 shutdown_connections 2024-04-23T18:12:24.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-23T18:12:24.423+0000 7f2ee51ea700 1 -- 172.21.15.64:0/1423747398 wait complete. 2024-04-23T18:12:25.140 DEBUG:teuthology.orchestra.run.smithi032:mds.a> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.a.service 2024-04-23T18:12:25.143 DEBUG:teuthology.orchestra.run.smithi032:mds.d> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.d.service 2024-04-23T18:12:25.187 DEBUG:teuthology.orchestra.run.smithi032:mds.g> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.g.service 2024-04-23T18:12:25.230 DEBUG:teuthology.orchestra.run.smithi032:mds.j> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.j.service 2024-04-23T18:12:25.273 DEBUG:teuthology.orchestra.run.smithi040:mds.b> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.b.service 2024-04-23T18:12:25.276 DEBUG:teuthology.orchestra.run.smithi040:mds.e> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.e.service 2024-04-23T18:12:25.320 DEBUG:teuthology.orchestra.run.smithi040:mds.h> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.h.service 2024-04-23T18:12:25.363 DEBUG:teuthology.orchestra.run.smithi040:mds.k> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.k.service 2024-04-23T18:12:25.406 DEBUG:teuthology.orchestra.run.smithi064:mds.c> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service 2024-04-23T18:12:25.409 DEBUG:teuthology.orchestra.run.smithi064:mds.f> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.f.service 2024-04-23T18:12:25.452 DEBUG:teuthology.orchestra.run.smithi064:mds.i> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.i.service 2024-04-23T18:12:25.495 DEBUG:teuthology.orchestra.run.smithi064:mds.l> sudo journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.l.service 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l;count:12 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:25.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:25.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:25 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.538 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-23T18:12:25.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:12:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l;count:12 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:25.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:25 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 -- 172.21.15.32:0/702452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102a00 msgr2=0x7f4b40109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 --2- 172.21.15.32:0/702452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102a00 0x7f4b40109320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009900 tx=0x7f4b3802a410 comp rx=0 tx=0).stop 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 -- 172.21.15.32:0/702452744 shutdown_connections 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 --2- 172.21.15.32:0/702452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102a00 0x7f4b40109320 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 --2- 172.21.15.32:0/702452744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40102040 0x7f4b401024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 --2- 172.21.15.32:0/702452744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40100e60 0x7f4b40101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 -- 172.21.15.32:0/702452744 >> 172.21.15.32:0/702452744 conn(0x7f4b400fc460 msgr2=0x7f4b400fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 -- 172.21.15.32:0/702452744 shutdown_connections 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.655+0000 7f4b482e8640 1 -- 172.21.15.32:0/702452744 wait complete. 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 Processor -- start 2024-04-23T18:12:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 -- start start 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40100e60 0x7f4b4006b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40102a00 0x7f4b401a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4010ca10 con 0x7f4b40102040 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4010c890 con 0x7f4b40100e60 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b482e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4010cb90 con 0x7f4b40102a00 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54392/0 (socket says 172.21.15.32:54392) 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 -- 172.21.15.32:0/1086208740 learned_addr learned my addr 172.21.15.32:0/1086208740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b46885640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40102a00 0x7f4b401a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b46084640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40100e60 0x7f4b4006b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 -- 172.21.15.32:0/1086208740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40102a00 msgr2=0x7f4b401a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40102a00 0x7f4b401a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 -- 172.21.15.32:0/1086208740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40100e60 msgr2=0x7f4b4006b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40100e60 0x7f4b4006b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.656+0000 7f4b45883640 1 -- 172.21.15.32:0/1086208740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b401a1ff0 con 0x7f4b40102040 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b45883640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000d950 tx=0x7f4b3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b30017030 con 0x7f4b40102040 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b401a2280 con 0x7f4b40102040 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b30004020 con 0x7f4b40102040 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300047a0 con 0x7f4b40102040 2024-04-23T18:12:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.657+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b401a27d0 con 0x7f4b40102040 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.658+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b40108200 con 0x7f4b40102040 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.659+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b30009a30 con 0x7f4b40102040 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.660+0000 7f4b377fe640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 0x7f4b2c0a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.660+0000 7f4b46084640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 0x7f4b2c0a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.660+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4b300cf890 con 0x7f4b40102040 2024-04-23T18:12:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.660+0000 7f4b46084640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 0x7f4b2c0a60a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4b40101bb0 tx=0x7f4b3c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:25.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.664+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b3009c710 con 0x7f4b40102040 2024-04-23T18:12:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:25.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l;count:12 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:25.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:25 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.854+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b4002d7e0 con 0x7f4b40102040 2024-04-23T18:12:25.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.855+0000 7f4b377fe640 1 -- 172.21.15.32:0/1086208740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18385 (secure 0 0 0) 0x7f4b3009c010 con 0x7f4b40102040 2024-04-23T18:12:25.850 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:25.850 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:12:15.803236+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 msgr2=0x7f4b2c0a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 0x7f4b2c0a60a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4b40101bb0 tx=0x7f4b3c009500 comp rx=0 tx=0).stop 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 msgr2=0x7f4b4019f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000d950 tx=0x7f4b3000de10 comp rx=0 tx=0).stop 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 shutdown_connections 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b40102a00 0x7f4b401a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b40102040 0x7f4b4019f2b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4b2c0a3be0 0x7f4b2c0a60a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 --2- 172.21.15.32:0/1086208740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b40100e60 0x7f4b4006b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.857+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 >> 172.21.15.32:0/1086208740 conn(0x7f4b400fc460 msgr2=0x7f4b401053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.858+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 shutdown_connections 2024-04-23T18:12:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.858+0000 7f4b482e8640 1 -- 172.21.15.32:0/1086208740 wait complete. 2024-04-23T18:12:25.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:12:25.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:12:25.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 -- 172.21.15.32:0/2117366025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea24101fe0 msgr2=0x7fea24102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/2117366025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea24101fe0 0x7fea24102460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea14009900 tx=0x7fea1402a410 comp rx=0 tx=0).stop 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 -- 172.21.15.32:0/2117366025 shutdown_connections 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/2117366025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea241029a0 0x7fea241092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/2117366025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea24101fe0 0x7fea24102460 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/2117366025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 0x7fea241012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 -- 172.21.15.32:0/2117366025 >> 172.21.15.32:0/2117366025 conn(0x7fea240fc420 msgr2=0x7fea240fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 -- 172.21.15.32:0/2117366025 shutdown_connections 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.974+0000 7fea2bf9b640 1 -- 172.21.15.32:0/2117366025 wait complete. 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 Processor -- start 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 -- start start 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 0x7fea24198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 0x7fea24198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410ca20 con 0x7fea241029a0 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410c8a0 con 0x7fea24101fe0 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2bf9b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410cba0 con 0x7fea24100e90 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54406/0 (socket says 172.21.15.32:54406) 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 -- 172.21.15.32:0/3732600316 learned_addr learned my addr 172.21.15.32:0/3732600316 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea29536640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 0x7fea24198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea29d37640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 0x7fea24198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 -- 172.21.15.32:0/3732600316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 msgr2=0x7fea24198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 0x7fea24198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 -- 172.21.15.32:0/3732600316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 msgr2=0x7fea24198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 0x7fea24198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea2a538640 1 -- 172.21.15.32:0/3732600316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2419fdf0 con 0x7fea241029a0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.975+0000 7fea29536640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 0x7fea24198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea2a538640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00eb10 tx=0x7fea1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c004b20 con 0x7fea241029a0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea241a00e0 con 0x7fea241029a0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea1c004cc0 con 0x7fea241029a0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c010670 con 0x7fea241029a0 2024-04-23T18:12:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.976+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea241a0600 con 0x7fea241029a0 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.977+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea241a08c0 con 0x7fea241029a0 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.978+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fea1c005000 con 0x7fea241029a0 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.978+0000 7fea1affd640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.978+0000 7fea29d37640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.979+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fea1c0d0670 con 0x7fea241029a0 2024-04-23T18:12:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.979+0000 7fea29d37640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 0x7fea100a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea20005d00 tx=0x7fea2000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:25.982+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea1c053ff0 con 0x7fea241029a0 2024-04-23T18:12:26.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.170+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fea241a0a90 con 0x7fea241029a0 2024-04-23T18:12:26.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.170+0000 7fea1affd640 1 -- 172.21.15.32:0/3732600316 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fea1c09d4f0 con 0x7fea241029a0 2024-04-23T18:12:26.164 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:12:26.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 msgr2=0x7fea100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 0x7fea100a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea20005d00 tx=0x7fea2000b410 comp rx=0 tx=0).stop 2024-04-23T18:12:26.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 msgr2=0x7fea2419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00eb10 tx=0x7fea1c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 shutdown_connections 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241029a0 0x7fea2419f780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea24101fe0 0x7fea24198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 --2- 172.21.15.32:0/3732600316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24100e90 0x7fea24198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 >> 172.21.15.32:0/3732600316 conn(0x7fea240fc420 msgr2=0x7fea240fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 shutdown_connections 2024-04-23T18:12:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.172+0000 7fea2bf9b640 1 -- 172.21.15.32:0/3732600316 wait complete. 2024-04-23T18:12:26.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:26.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 -- 172.21.15.32:0/434089896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 msgr2=0x7f8014103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 --2- 172.21.15.32:0/434089896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014103620 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8008009900 tx=0x7f800802a410 comp rx=0 tx=0).stop 2024-04-23T18:12:26.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 -- 172.21.15.32:0/434089896 shutdown_connections 2024-04-23T18:12:26.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 --2- 172.21.15.32:0/434089896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014103b60 0x7f801410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 --2- 172.21.15.32:0/434089896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014103620 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 --2- 172.21.15.32:0/434089896 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014101fb0 0x7f80141023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 -- 172.21.15.32:0/434089896 >> 172.21.15.32:0/434089896 conn(0x7f80140fd540 msgr2=0x7f80140ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.285+0000 7f80189cd640 1 -- 172.21.15.32:0/434089896 shutdown_connections 2024-04-23T18:12:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 -- 172.21.15.32:0/434089896 wait complete. 2024-04-23T18:12:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 Processor -- start 2024-04-23T18:12:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 -- start start 2024-04-23T18:12:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014101fb0 0x7f8014198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 0x7f801419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f801410db50 con 0x7f80141031a0 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f801410d9d0 con 0x7f8014101fb0 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f80189cd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f801410dcd0 con 0x7f8014103b60 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.286+0000 7f8011d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54428/0 (socket says 172.21.15.32:54428) 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 -- 172.21.15.32:0/4189014436 learned_addr learned my addr 172.21.15.32:0/4189014436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8012d9d640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 0x7f801419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 -- 172.21.15.32:0/4189014436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 msgr2=0x7f801419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 0x7f801419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f801259c640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014101fb0 0x7f8014198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 -- 172.21.15.32:0/4189014436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014101fb0 msgr2=0x7f8014198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014101fb0 0x7f8014198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 -- 172.21.15.32:0/4189014436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801419fe50 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8012d9d640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 0x7f801419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f8011d9b640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8008005940 tx=0x7f8008004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800803f030 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8008003cb0 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80141a00e0 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008035070 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.287+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80141a0650 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.288+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8014109350 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.289+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8008042030 con 0x7f80141031a0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.290+0000 7f80037fe640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 0x7f7ffc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.290+0000 7f801259c640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 0x7f7ffc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.290+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f80080f0a00 con 0x7f80141031a0 2024-04-23T18:12:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.290+0000 7f801259c640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 0x7f7ffc0a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8014103000 tx=0x7f80040072f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.295+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8008003e60 con 0x7f80141031a0 2024-04-23T18:12:26.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:26 smithi032 ceph-mon[29221]: Deploying daemon mds.l on smithi064 2024-04-23T18:12:26.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:26 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1086208740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:26.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:26 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3732600316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.519+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8014062b50 con 0x7f80141031a0 2024-04-23T18:12:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.520+0000 7f80037fe640 1 -- 172.21.15.32:0/4189014436 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f80080bd7e0 con 0x7f80141031a0 2024-04-23T18:12:26.514 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:26.514 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-23T18:12:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 msgr2=0x7f7ffc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 0x7f7ffc0a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8014103000 tx=0x7f80040072f0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 msgr2=0x7f8014198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8008005940 tx=0x7f8008004430 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 shutdown_connections 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8014103b60 0x7f801419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80141031a0 0x7f8014198770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7ffc0a3bb0 0x7f7ffc0a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 --2- 172.21.15.32:0/4189014436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8014101fb0 0x7f8014198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 >> 172.21.15.32:0/4189014436 conn(0x7f80140fd540 msgr2=0x7f8014106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 shutdown_connections 2024-04-23T18:12:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:26.522+0000 7f80189cd640 1 -- 172.21.15.32:0/4189014436 wait complete. 2024-04-23T18:12:26.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:26 smithi064 ceph-mon[33062]: Deploying daemon mds.l on smithi064 2024-04-23T18:12:26.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:26 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1086208740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:26.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:26 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3732600316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:26.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:26 smithi040 ceph-mon[32626]: Deploying daemon mds.l on smithi064 2024-04-23T18:12:26.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:26 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1086208740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:12:26.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:26 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3732600316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:12:27.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:27 smithi064 ceph-mon[33062]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:27.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:27 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4189014436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:27.673 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:27 smithi064 systemd[1]: Starting Ceph mds.l for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:27.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:27 smithi032 ceph-mon[29221]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:27.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:27 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4189014436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:27 smithi040 ceph-mon[32626]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:27 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4189014436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:27.997 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:27 smithi064 podman[63604]: 2024-04-23 18:12:27.615220468 +0000 UTC m=+0.029836351 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:27.997 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:27 smithi064 podman[63604]: 2024-04-23 18:12:27.734750004 +0000 UTC m=+0.149365909 container create 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-23T18:12:27.997 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 podman[63604]: 2024-04-23 18:12:28.003876909 +0000 UTC m=+0.418492774 container init 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:12:28.422 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 podman[63604]: 2024-04-23 18:12:28.00668426 +0000 UTC m=+0.421300132 container start 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-23T18:12:28.422 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 ceph-mds[63623]: -- 172.21.15.64:0/348107613 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e3bc09a40 con 0x556e3bbdc800 2024-04-23T18:12:28.422 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l[63619]: starting mds.l at 2024-04-23T18:12:28.423 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 bash[63604]: 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb 2024-04-23T18:12:28.423 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:12:28 smithi064 systemd[1]: Started Ceph mds.l for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:28.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 -- 172.21.15.32:0/146194654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec103b60 msgr2=0x7f2aec10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 --2- 172.21.15.32:0/146194654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec103b60 0x7f2aec10a3f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009ca0 tx=0x7f2ae402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 -- 172.21.15.32:0/146194654 shutdown_connections 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 --2- 172.21.15.32:0/146194654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec103b60 0x7f2aec10a3f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 --2- 172.21.15.32:0/146194654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec1031a0 0x7f2aec103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 --2- 172.21.15.32:0/146194654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec101fb0 0x7f2aec1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.652+0000 7f2af3544640 1 -- 172.21.15.32:0/146194654 >> 172.21.15.32:0/146194654 conn(0x7f2aec0fd540 msgr2=0x7f2aec0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- 172.21.15.32:0/146194654 shutdown_connections 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- 172.21.15.32:0/146194654 wait complete. 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 Processor -- start 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- start start 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec1031a0 0x7f2aec1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec103b60 0x7f2aec19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec10daf0 con 0x7f2aec101fb0 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec10d970 con 0x7f2aec1031a0 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af3544640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec10dc70 con 0x7f2aec103b60 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af12e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af12e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33970/0 (socket says 172.21.15.32:33970) 2024-04-23T18:12:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.653+0000 7f2af12e0640 1 -- 172.21.15.32:0/274127931 learned_addr learned my addr 172.21.15.32:0/274127931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af0adf640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec1031a0 0x7f2aec1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af1ae1640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec103b60 0x7f2aec19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 -- 172.21.15.32:0/274127931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec103b60 msgr2=0x7f2aec19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec103b60 0x7f2aec19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 -- 172.21.15.32:0/274127931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec1031a0 msgr2=0x7f2aec1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec1031a0 0x7f2aec1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 -- 172.21.15.32:0/274127931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec19fda0 con 0x7f2aec101fb0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af12e0640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9c0 tx=0x7f2ae800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800cd80 con 0x7f2aec101fb0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ae80035f0 con 0x7f2aec101fb0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aec1a0090 con 0x7f2aec101fb0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8010690 con 0x7f2aec101fb0 2024-04-23T18:12:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.654+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aec1a0630 con 0x7f2aec101fb0 2024-04-23T18:12:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.655+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aec109350 con 0x7f2aec101fb0 2024-04-23T18:12:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.656+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2ae80037a0 con 0x7f2aec101fb0 2024-04-23T18:12:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.657+0000 7f2ae27fc640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 0x7f2ad80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.657+0000 7f2af0adf640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 0x7f2ad80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.657+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2ae80d03f0 con 0x7f2aec101fb0 2024-04-23T18:12:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.658+0000 7f2af0adf640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 0x7f2ad80a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2aec199690 tx=0x7f2adc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.661+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ae809d1d0 con 0x7f2aec101fb0 2024-04-23T18:12:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.885+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2aec02d5c0 con 0x7f2aec101fb0 2024-04-23T18:12:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.885+0000 7f2ae27fc640 1 -- 172.21.15.32:0/274127931 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f2ae809cad0 con 0x7f2aec101fb0 2024-04-23T18:12:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:28.880 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:12:28.880 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 msgr2=0x7f2ad80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 0x7f2ad80a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2aec199690 tx=0x7f2adc005f90 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 msgr2=0x7f2aec198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9c0 tx=0x7f2ae800ee80 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 shutdown_connections 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2aec103b60 0x7f2aec19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2ad80a3c30 0x7f2ad80a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aec1031a0 0x7f2aec1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 --2- 172.21.15.32:0/274127931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec101fb0 0x7f2aec198180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 >> 172.21.15.32:0/274127931 conn(0x7f2aec0fd540 msgr2=0x7f2aec106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 shutdown_connections 2024-04-23T18:12:28.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:28.887+0000 7f2af3544640 1 -- 172.21.15.32:0/274127931 wait complete. 2024-04-23T18:12:29.091 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6832/2951030033,v1:172.21.15.64:6833/2951030033] up:boot 2024-04-23T18:12:29.091 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: fsmap 1 up:standby 2024-04-23T18:12:29.091 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:12:29.091 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: Deploying daemon mds.k on smithi040 2024-04-23T18:12:29.092 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:29 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/274127931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6832/2951030033,v1:172.21.15.64:6833/2951030033] up:boot 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: fsmap 1 up:standby 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:29.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: Deploying daemon mds.k on smithi040 2024-04-23T18:12:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:29 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/274127931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:29.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6832/2951030033,v1:172.21.15.64:6833/2951030033] up:boot 2024-04-23T18:12:29.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: fsmap 1 up:standby 2024-04-23T18:12:29.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:12:29.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: Deploying daemon mds.k on smithi040 2024-04-23T18:12:29.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:29 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/274127931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:30.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:30.921 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:30 smithi040 podman[61860]: 2024-04-23 18:12:30.798153489 +0000 UTC m=+0.018587736 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:31.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 -- 172.21.15.32:0/3414543028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 msgr2=0x7f8c6810a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/3414543028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6810a4a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8c60009cd0 tx=0x7f8c6002a8e0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 -- 172.21.15.32:0/3414543028 shutdown_connections 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/3414543028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6810a4a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/3414543028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68103250 0x7f8c681036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/3414543028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68100280 0x7f8c681006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 -- 172.21.15.32:0/3414543028 >> 172.21.15.32:0/3414543028 conn(0x7f8c680fbbb0 msgr2=0x7f8c680fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 -- 172.21.15.32:0/3414543028 shutdown_connections 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.014+0000 7f8c6fb19640 1 -- 172.21.15.32:0/3414543028 wait complete. 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 Processor -- start 2024-04-23T18:12:31.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 -- start start 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68100280 0x7f8c681981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68103250 0x7f8c681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6810db20 con 0x7f8c68103c10 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6810d9a0 con 0x7f8c68100280 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6fb19640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6810dca0 con 0x7f8c68103250 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6e0b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6d0b4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68103250 0x7f8c681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6e0b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33994/0 (socket says 172.21.15.32:33994) 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6e0b6640 1 -- 172.21.15.32:0/1189552177 learned_addr learned my addr 172.21.15.32:0/1189552177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.015+0000 7f8c6d8b5640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68100280 0x7f8c681981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 -- 172.21.15.32:0/1189552177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68103250 msgr2=0x7f8c681986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68103250 0x7f8c681986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 -- 172.21.15.32:0/1189552177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68100280 msgr2=0x7f8c681981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68100280 0x7f8c681981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 -- 172.21.15.32:0/1189552177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6819fdd0 con 0x7f8c68103c10 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6e0b6640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c600318b0 tx=0x7f8c60003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60004040 con 0x7f8c68103c10 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c681a0060 con 0x7f8c68103c10 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c60004690 con 0x7f8c68103c10 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6003b400 con 0x7f8c68103c10 2024-04-23T18:12:31.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.016+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c681a05b0 con 0x7f8c68103c10 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.017+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c68109380 con 0x7f8c68103c10 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.018+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8c60030630 con 0x7f8c68103c10 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.019+0000 7f8c5effd640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 0x7f8c540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.019+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8c600f0fc0 con 0x7f8c68103c10 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.019+0000 7f8c6d8b5640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 0x7f8c540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:31.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.019+0000 7f8c6d8b5640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 0x7f8c540a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c6806a460 tx=0x7f8c64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:31.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.024+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c600bde40 con 0x7f8c68103c10 2024-04-23T18:12:31.190 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:31 smithi040 ceph-mon[32626]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:31.190 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:30 smithi040 podman[61860]: 2024-04-23 18:12:30.927222625 +0000 UTC m=+0.147656894 container create 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:12:31.190 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 podman[61860]: 2024-04-23 18:12:31.196874616 +0000 UTC m=+0.417308844 container init 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T18:12:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.251+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c68062b50 con 0x7f8c68103c10 2024-04-23T18:12:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.251+0000 7f8c5effd640 1 -- 172.21.15.32:0/1189552177 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f8c600098b0 con 0x7f8c68103c10 2024-04-23T18:12:31.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:31.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:12:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 msgr2=0x7f8c540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 0x7f8c540a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c6806a460 tx=0x7f8c64005f90 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 msgr2=0x7f8c6819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c600318b0 tx=0x7f8c60003b80 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 shutdown_connections 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c68103c10 0x7f8c6819f760 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c68103250 0x7f8c681986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8c540a3bb0 0x7f8c540a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 --2- 172.21.15.32:0/1189552177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c68100280 0x7f8c681981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.253+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 >> 172.21.15.32:0/1189552177 conn(0x7f8c680fbbb0 msgr2=0x7f8c68106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.254+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 shutdown_connections 2024-04-23T18:12:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:31.254+0000 7f8c6fb19640 1 -- 172.21.15.32:0/1189552177 wait complete. 2024-04-23T18:12:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:31 smithi032 ceph-mon[29221]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:31 smithi064 ceph-mon[33062]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:31.617 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 podman[61860]: 2024-04-23 18:12:31.199878062 +0000 UTC m=+0.420312302 container start 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:12:31.617 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 ceph-mds[61879]: -- 172.21.15.40:0/2212512540 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb22b8ba40 con 0x55fb22b5ec00 2024-04-23T18:12:31.617 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k[61875]: starting mds.k at 2024-04-23T18:12:31.617 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 bash[61860]: 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 2024-04-23T18:12:31.617 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:12:31 smithi040 systemd[1]: Started Ceph mds.k for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:32.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1189552177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:32.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:32.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:32.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:32.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:32 smithi032 ceph-mon[29221]: Deploying daemon mds.j on smithi032 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1189552177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:32 smithi040 ceph-mon[32626]: Deploying daemon mds.j on smithi032 2024-04-23T18:12:32.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1189552177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:32.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:32.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:32.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:32 smithi064 ceph-mon[33062]: Deploying daemon mds.j on smithi032 2024-04-23T18:12:33.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:33 smithi040 ceph-mon[32626]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:33 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6832/768829407,v1:172.21.15.40:6833/768829407] up:boot 2024-04-23T18:12:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:33 smithi040 ceph-mon[32626]: fsmap 2 up:standby 2024-04-23T18:12:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:33 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:12:33.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:33 smithi032 ceph-mon[29221]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:33.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:33 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6832/768829407,v1:172.21.15.40:6833/768829407] up:boot 2024-04-23T18:12:33.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:33 smithi032 ceph-mon[29221]: fsmap 2 up:standby 2024-04-23T18:12:33.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:33 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:12:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:33 smithi064 ceph-mon[33062]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:33 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6832/768829407,v1:172.21.15.40:6833/768829407] up:boot 2024-04-23T18:12:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:33 smithi064 ceph-mon[33062]: fsmap 2 up:standby 2024-04-23T18:12:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:33 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:12:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 -- 172.21.15.32:0/2663297268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0102040 msgr2=0x7f46a01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2663297268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0102040 0x7f46a01024c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f469c009900 tx=0x7f469c02a410 comp rx=0 tx=0).stop 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 -- 172.21.15.32:0/2663297268 shutdown_connections 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2663297268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 0x7f46a0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2663297268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0102040 0x7f46a01024c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2663297268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0100e60 0x7f46a0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 -- 172.21.15.32:0/2663297268 >> 172.21.15.32:0/2663297268 conn(0x7f46a00fc420 msgr2=0x7f46a00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 -- 172.21.15.32:0/2663297268 shutdown_connections 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 -- 172.21.15.32:0/2663297268 wait complete. 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.511+0000 7f46a80cf640 1 Processor -- start 2024-04-23T18:12:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 -- start start 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0102040 0x7f46a019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 0x7f46a019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f46a010ca00 con 0x7f46a0100e60 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46a010c880 con 0x7f46a0102040 2024-04-23T18:12:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f46a010cb80 con 0x7f46a0102a00 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34002/0 (socket says 172.21.15.32:34002) 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 -- 172.21.15.32:0/2193602571 learned_addr learned my addr 172.21.15.32:0/2193602571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a566a640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0102040 0x7f46a019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a666c640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 0x7f46a019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 -- 172.21.15.32:0/2193602571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 msgr2=0x7f46a019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 0x7f46a019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 -- 172.21.15.32:0/2193602571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0102040 msgr2=0x7f46a019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0102040 0x7f46a019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 -- 172.21.15.32:0/2193602571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a019fe20 con 0x7f46a0100e60 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a5e6b640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f469000e9c0 tx=0x7f469000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469000cd80 con 0x7f46a0100e60 2024-04-23T18:12:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46900035f0 con 0x7f46a0100e60 2024-04-23T18:12:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a01a00b0 con 0x7f46a0100e60 2024-04-23T18:12:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.512+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690010690 con 0x7f46a0100e60 2024-04-23T18:12:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.513+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a01a0620 con 0x7f46a0100e60 2024-04-23T18:12:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.514+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a0108200 con 0x7f46a0100e60 2024-04-23T18:12:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.515+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f46900037a0 con 0x7f46a0100e60 2024-04-23T18:12:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.516+0000 7f4696ffd640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.516+0000 7f46a566a640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.516+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f46900d0400 con 0x7f46a0100e60 2024-04-23T18:12:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.517+0000 7f46a566a640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f469c000c00 tx=0x7f469c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.520+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f469009d280 con 0x7f46a0100e60 2024-04-23T18:12:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.759+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46a002d5c0 con 0x7f46a0100e60 2024-04-23T18:12:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.760+0000 7f4696ffd640 1 -- 172.21.15.32:0/2193602571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2215 (secure 0 0 0) 0x7f469009cb80 con 0x7f46a0100e60 2024-04-23T18:12:33.754 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:33.754 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:12:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 msgr2=0x7f468c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f469c000c00 tx=0x7f469c0314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 msgr2=0x7f46a0069dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f469000e9c0 tx=0x7f469000ee80 comp rx=0 tx=0).stop 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 shutdown_connections 2024-04-23T18:12:33.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a0102a00 0x7f46a019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a0102040 0x7f46a019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 --2- 172.21.15.32:0/2193602571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a0100e60 0x7f46a0069dc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 >> 172.21.15.32:0/2193602571 conn(0x7f46a00fc420 msgr2=0x7f46a01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 shutdown_connections 2024-04-23T18:12:33.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:33.763+0000 7f46a80cf640 1 -- 172.21.15.32:0/2193602571 wait complete. 2024-04-23T18:12:33.758 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:33 smithi032 systemd[1]: Starting Ceph mds.j for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:34.110 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:33 smithi032 podman[69958]: 2024-04-23 18:12:33.760855725 +0000 UTC m=+0.021318070 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:34.110 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:33 smithi032 podman[69958]: 2024-04-23 18:12:33.872199792 +0000 UTC m=+0.132662122 container create 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-23T18:12:34.110 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 podman[69958]: 2024-04-23 18:12:34.116518734 +0000 UTC m=+0.376981064 container init 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T18:12:34.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:34 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2193602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:34.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:34 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2193602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:34.508 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 podman[69958]: 2024-04-23 18:12:34.120302706 +0000 UTC m=+0.380765039 container start 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T18:12:34.508 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 ceph-mds[69979]: -- 172.21.15.32:0/780195662 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55777de0da40 con 0x55777dde1000 2024-04-23T18:12:34.508 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j[69975]: starting mds.j at 2024-04-23T18:12:34.508 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 bash[69958]: 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f 2024-04-23T18:12:34.508 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:12:34 smithi032 systemd[1]: Started Ceph mds.j for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:34.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:34 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2193602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:35.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: Deploying daemon mds.d on smithi032 2024-04-23T18:12:35.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.32:6834/1839255789,v1:172.21.15.32:6835/1839255789] up:boot 2024-04-23T18:12:35.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: fsmap 3 up:standby 2024-04-23T18:12:35.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:35 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:35.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:35.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: Deploying daemon mds.d on smithi032 2024-04-23T18:12:35.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.32:6834/1839255789,v1:172.21.15.32:6835/1839255789] up:boot 2024-04-23T18:12:35.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: fsmap 3 up:standby 2024-04-23T18:12:35.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:35 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:12:35.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:35.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: Deploying daemon mds.d on smithi032 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.32:6834/1839255789,v1:172.21.15.32:6835/1839255789] up:boot 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: fsmap 3 up:standby 2024-04-23T18:12:35.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:35 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:12:35.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- 172.21.15.32:0/4036267090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c100e60 msgr2=0x7f3e5c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4036267090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c100e60 0x7f3e5c101280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009900 tx=0x7f3e4c02a410 comp rx=0 tx=0).stop 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- 172.21.15.32:0/4036267090 shutdown_connections 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4036267090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4036267090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c102040 0x7f3e5c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4036267090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c100e60 0x7f3e5c101280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- 172.21.15.32:0/4036267090 >> 172.21.15.32:0/4036267090 conn(0x7f3e5c0fc460 msgr2=0x7f3e5c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- 172.21.15.32:0/4036267090 shutdown_connections 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- 172.21.15.32:0/4036267090 wait complete. 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 Processor -- start 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 -- start start 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.895+0000 7f3e619f9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 0x7f3e5c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c1a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10ca70 con 0x7f3e5c102040 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10c8f0 con 0x7f3e5c102a00 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10cbf0 con 0x7f3e5c100e60 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e5affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 0x7f3e5c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34052/0 (socket says 172.21.15.32:34052) 2024-04-23T18:12:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 -- 172.21.15.32:0/4208674108 learned_addr learned my addr 172.21.15.32:0/4208674108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e5b7fe640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c1a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 -- 172.21.15.32:0/4208674108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 msgr2=0x7f3e5c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 0x7f3e5c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 -- 172.21.15.32:0/4208674108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 msgr2=0x7f3e5c1a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c1a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 -- 172.21.15.32:0/4208674108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c1a2080 con 0x7f3e5c102040 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e5affd640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 0x7f3e5c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e5b7fe640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c1a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e53fff640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400e9f0 tx=0x7f3e4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400cd80 con 0x7f3e5c102040 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e440054d0 con 0x7f3e5c102040 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c1a2370 con 0x7f3e5c102040 2024-04-23T18:12:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44010690 con 0x7f3e5c102040 2024-04-23T18:12:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.896+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c1a2940 con 0x7f3e5c102040 2024-04-23T18:12:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.897+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c1a2c00 con 0x7f3e5c102040 2024-04-23T18:12:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.898+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3e44004e90 con 0x7f3e5c102040 2024-04-23T18:12:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.899+0000 7f3e609f7640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 0x7f3e480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.899+0000 7f3e5affd640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 0x7f3e480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.899+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3e440d03a0 con 0x7f3e5c102040 2024-04-23T18:12:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.900+0000 7f3e5affd640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 0x7f3e480a60f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009230 tx=0x7f3e4c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:35.904+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e4409d220 con 0x7f3e5c102040 2024-04-23T18:12:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.135+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e5c1a2dd0 con 0x7f3e5c102040 2024-04-23T18:12:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.135+0000 7f3e609f7640 1 -- 172.21.15.32:0/4208674108 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7f3e4409cb20 con 0x7f3e5c102040 2024-04-23T18:12:36.129 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:36.130 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:12:36.130 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 4 2024-04-23T18:12:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 msgr2=0x7f3e480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 0x7f3e480a60f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009230 tx=0x7f3e4c0314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 msgr2=0x7f3e5c19a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400e9f0 tx=0x7f3e4400edf0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 shutdown_connections 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3e480a3c30 0x7f3e480a60f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e5c102a00 0x7f3e5c1a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5c102040 0x7f3e5c19a9a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 --2- 172.21.15.32:0/4208674108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e5c100e60 0x7f3e5c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 >> 172.21.15.32:0/4208674108 conn(0x7f3e5c0fc460 msgr2=0x7f3e5c06a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 shutdown_connections 2024-04-23T18:12:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:36.137+0000 7f3e619f9640 1 -- 172.21.15.32:0/4208674108 wait complete. 2024-04-23T18:12:36.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:36 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4208674108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:36.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:36 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4208674108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:36.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:36 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4208674108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:36.758 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:36 smithi032 systemd[1]: Starting Ceph mds.d for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:37.087 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:36 smithi032 podman[70312]: 2024-04-23 18:12:36.703678211 +0000 UTC m=+0.018551522 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:37.087 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:36 smithi032 podman[70312]: 2024-04-23 18:12:36.840077798 +0000 UTC m=+0.154951090 container create 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:12:37.088 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 podman[70312]: 2024-04-23 18:12:37.09354859 +0000 UTC m=+0.408421895 container init 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T18:12:37.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:37 smithi032 ceph-mon[29221]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:37.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:37.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:37 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:37.508 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 podman[70312]: 2024-04-23 18:12:37.096440826 +0000 UTC m=+0.411314125 container start 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:12:37.508 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 ceph-mds[70331]: -- 172.21.15.32:0/2104043982 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec3063ba40 con 0x55ec3060f000 2024-04-23T18:12:37.509 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d[70327]: starting mds.d at 2024-04-23T18:12:37.509 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 bash[70312]: 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe 2024-04-23T18:12:37.509 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:12:37 smithi032 systemd[1]: Started Ceph mds.d for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:37.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:37 smithi040 ceph-mon[32626]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:37.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:37 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:37.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:37 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:37.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:37 smithi064 ceph-mon[33062]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:37.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:37 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:37.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:37 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:38.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:38.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- 172.21.15.32:0/3573732827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210105360 msgr2=0x7fe210107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 --2- 172.21.15.32:0/3573732827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210105360 0x7fe210107750 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe200009900 tx=0x7fe20002a410 comp rx=0 tx=0).stop 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- 172.21.15.32:0/3573732827 shutdown_connections 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 --2- 172.21.15.32:0/3573732827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210107c90 0x7fe21010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 --2- 172.21.15.32:0/3573732827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210105360 0x7fe210107750 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 --2- 172.21.15.32:0/3573732827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210102a30 0x7fe210104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- 172.21.15.32:0/3573732827 >> 172.21.15.32:0/3573732827 conn(0x7fe2100fc460 msgr2=0x7fe2100fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- 172.21.15.32:0/3573732827 shutdown_connections 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- 172.21.15.32:0/3573732827 wait complete. 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 Processor -- start 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.268+0000 7fe21647c640 1 -- start start 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210105360 0x7fe210198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 0x7fe21019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010dab0 con 0x7fe210107c90 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010d930 con 0x7fe210102a30 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe21647c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010dc30 con 0x7fe210105360 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe214a19640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 0x7fe21019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:34436/0 (socket says 172.21.15.32:34436) 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 -- 172.21.15.32:0/4129543662 learned_addr learned my addr 172.21.15.32:0/4129543662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20f7fe640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210105360 0x7fe210198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 -- 172.21.15.32:0/4129543662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210105360 msgr2=0x7fe210198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210105360 0x7fe210198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 -- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 msgr2=0x7fe21019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 0x7fe21019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe20ffff640 1 -- 172.21.15.32:0/4129543662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21019fe50 con 0x7fe210102a30 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.269+0000 7fe214a19640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 0x7fe21019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe20ffff640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe204009b60 tx=0x7fe204007590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204009d70 con 0x7fe210102a30 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe204007c40 con 0x7fe210102a30 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe210100880 con 0x7fe210102a30 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204005280 con 0x7fe210102a30 2024-04-23T18:12:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.270+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe210100da0 con 0x7fe210102a30 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.271+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2100fdf50 con 0x7fe210102a30 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.272+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe204005420 con 0x7fe210102a30 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.273+0000 7fe20d7fa640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 0x7fe1f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.273+0000 7fe20f7fe640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 0x7fe1f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.273+0000 7fe20f7fe640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 0x7fe1f80a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe200009230 tx=0x7fe2000314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.274+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe2040d1520 con 0x7fe210102a30 2024-04-23T18:12:38.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.279+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe204007df0 con 0x7fe210102a30 2024-04-23T18:12:38.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.505+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe21002d5c0 con 0x7fe210102a30 2024-04-23T18:12:38.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.506+0000 7fe20d7fa640 1 -- 172.21.15.32:0/4129543662 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7fe20409fca0 con 0x7fe210102a30 2024-04-23T18:12:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:38.500 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:12:38.500 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 5 2024-04-23T18:12:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 msgr2=0x7fe1f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 0x7fe1f80a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe200009230 tx=0x7fe2000314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 msgr2=0x7fe210198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe204009b60 tx=0x7fe204007590 comp rx=0 tx=0).stop 2024-04-23T18:12:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 shutdown_connections 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210107c90 0x7fe21019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe210105360 0x7fe210198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe1f80a3bb0 0x7fe1f80a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 --2- 172.21.15.32:0/4129543662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe210102a30 0x7fe210198230 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 >> 172.21.15.32:0/4129543662 conn(0x7fe2100fc460 msgr2=0x7fe2101048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 shutdown_connections 2024-04-23T18:12:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:38.508+0000 7fe21647c640 1 -- 172.21.15.32:0/4129543662 wait complete. 2024-04-23T18:12:38.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.32:6836/2912347592,v1:172.21.15.32:6837/2912347592] up:boot 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: fsmap 4 up:standby 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:38.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:38 smithi032 ceph-mon[29221]: Deploying daemon mds.e on smithi040 2024-04-23T18:12:38.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.32:6836/2912347592,v1:172.21.15.32:6837/2912347592] up:boot 2024-04-23T18:12:38.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: fsmap 4 up:standby 2024-04-23T18:12:38.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:12:38.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:38.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:38.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:38.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:38.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:38 smithi040 ceph-mon[32626]: Deploying daemon mds.e on smithi040 2024-04-23T18:12:38.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.32:6836/2912347592,v1:172.21.15.32:6837/2912347592] up:boot 2024-04-23T18:12:38.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: fsmap 4 up:standby 2024-04-23T18:12:38.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:12:38.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:38.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:38.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:38.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:38.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:38 smithi064 ceph-mon[33062]: Deploying daemon mds.e on smithi040 2024-04-23T18:12:39.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:39 smithi040 ceph-mon[32626]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:39.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:39 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4129543662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:39 smithi064 ceph-mon[33062]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:39.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:39 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4129543662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:39.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:39 smithi032 ceph-mon[29221]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:39.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:39 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4129543662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:40.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:40.617 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:12:40 smithi040 systemd[1]: Started Ceph mds.e for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 -- 172.21.15.32:0/2704626620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0102040 msgr2=0x7f7af01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 --2- 172.21.15.32:0/2704626620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0102040 0x7f7af01024c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009900 tx=0x7f7ad802a410 comp rx=0 tx=0).stop 2024-04-23T18:12:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 -- 172.21.15.32:0/2704626620 shutdown_connections 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 --2- 172.21.15.32:0/2704626620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 0x7f7af0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 --2- 172.21.15.32:0/2704626620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0102040 0x7f7af01024c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 --2- 172.21.15.32:0/2704626620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0100e60 0x7f7af0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 -- 172.21.15.32:0/2704626620 >> 172.21.15.32:0/2704626620 conn(0x7f7af00fc460 msgr2=0x7f7af00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 -- 172.21.15.32:0/2704626620 shutdown_connections 2024-04-23T18:12:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.632+0000 7f7af5091640 1 -- 172.21.15.32:0/2704626620 wait complete. 2024-04-23T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 Processor -- start 2024-04-23T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 -- start start 2024-04-23T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0102040 0x7f7af01034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 0x7f7af01039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010ca10 con 0x7f7af0100e60 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010c890 con 0x7f7af0102a00 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7af5091640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010cb90 con 0x7f7af0102040 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7aeed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7aeed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51386/0 (socket says 172.21.15.32:51386) 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7aeed9d640 1 -- 172.21.15.32:0/94251326 learned_addr learned my addr 172.21.15.32:0/94251326 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7aee59c640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0102040 0x7f7af01034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.633+0000 7f7aef59e640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 0x7f7af01039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 -- 172.21.15.32:0/94251326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0102040 msgr2=0x7f7af01034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0102040 0x7f7af01034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 -- 172.21.15.32:0/94251326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 msgr2=0x7f7af01039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 0x7f7af01039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 -- 172.21.15.32:0/94251326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0104280 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7aeed9d640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e9c0 tx=0x7f7ae000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae000cd80 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af01a2300 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae00035f0 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0010690 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.634+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01a2820 con 0x7f7af0100e60 2024-04-23T18:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.635+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af0069250 con 0x7f7af0100e60 2024-04-23T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.636+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7ae0004000 con 0x7f7af0100e60 2024-04-23T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.637+0000 7f7ae7fff640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 0x7f7adc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.637+0000 7f7aee59c640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 0x7f7adc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.637+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7ae00d04e0 con 0x7f7af0100e60 2024-04-23T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.638+0000 7f7aee59c640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 0x7f7adc0a60f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000c00 tx=0x7f7ad80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.642+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ae009d2c0 con 0x7f7af0100e60 2024-04-23T18:12:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.869+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7af01a2d00 con 0x7f7af0100e60 2024-04-23T18:12:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.870+0000 7f7ae7fff640 1 -- 172.21.15.32:0/94251326 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f7ae009cbc0 con 0x7f7af0100e60 2024-04-23T18:12:40.864 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:40.865 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:12:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 6 2024-04-23T18:12:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 msgr2=0x7f7adc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 0x7f7adc0a60f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000c00 tx=0x7f7ad80314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 msgr2=0x7f7af0104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e9c0 tx=0x7f7ae000ee80 comp rx=0 tx=0).stop 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 shutdown_connections 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7adc0a3c30 0x7f7adc0a60f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0102a00 0x7f7af01039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7af0102040 0x7f7af01034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 --2- 172.21.15.32:0/94251326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af0100e60 0x7f7af0104e70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.872+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 >> 172.21.15.32:0/94251326 conn(0x7f7af00fc460 msgr2=0x7f7af00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.873+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 shutdown_connections 2024-04-23T18:12:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:40.873+0000 7f7af5091640 1 -- 172.21.15.32:0/94251326 wait complete. 2024-04-23T18:12:41.537 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:41.537 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6834/1068324383,v1:172.21.15.40:6835/1068324383] up:boot 2024-04-23T18:12:41.537 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: fsmap 5 up:standby 2024-04-23T18:12:41.537 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: Deploying daemon mds.f on smithi064 2024-04-23T18:12:41.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:41 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/94251326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6834/1068324383,v1:172.21.15.40:6835/1068324383] up:boot 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: fsmap 5 up:standby 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: Deploying daemon mds.f on smithi064 2024-04-23T18:12:41.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:41 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/94251326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:41.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:41.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6834/1068324383,v1:172.21.15.40:6835/1068324383] up:boot 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: fsmap 5 up:standby 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: Deploying daemon mds.f on smithi064 2024-04-23T18:12:41.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:41 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/94251326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:42.672 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:42 smithi064 systemd[1]: Starting Ceph mds.f for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:42.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:42.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 -- 172.21.15.32:0/601129600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370100f90 msgr2=0x7f83701013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 --2- 172.21.15.32:0/601129600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370100f90 0x7f83701013b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8360009900 tx=0x7f836002a410 comp rx=0 tx=0).stop 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 -- 172.21.15.32:0/601129600 shutdown_connections 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 --2- 172.21.15.32:0/601129600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370102b40 0x7f83701093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 --2- 172.21.15.32:0/601129600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 0x7f8370102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 --2- 172.21.15.32:0/601129600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370100f90 0x7f83701013b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 -- 172.21.15.32:0/601129600 >> 172.21.15.32:0/601129600 conn(0x7f83700fc520 msgr2=0x7f83700fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 -- 172.21.15.32:0/601129600 shutdown_connections 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.994+0000 7f8376507640 1 -- 172.21.15.32:0/601129600 wait complete. 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 Processor -- start 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 -- start start 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370100f90 0x7f8370198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 0x7f8370198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:42.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f837010c9f0 con 0x7f8370102b40 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f837010c870 con 0x7f8370100f90 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8376507640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f837010cb70 con 0x7f8370102180 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51416/0 (socket says 172.21.15.32:51416) 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 -- 172.21.15.32:0/1144433186 learned_addr learned my addr 172.21.15.32:0/1144433186 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f836f7fe640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 0x7f8370198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f836ffff640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370100f90 0x7f8370198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 -- 172.21.15.32:0/1144433186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 msgr2=0x7f8370198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 0x7f8370198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 -- 172.21.15.32:0/1144433186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370100f90 msgr2=0x7f8370198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370100f90 0x7f8370198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.995+0000 7f8374aa4640 1 -- 172.21.15.32:0/1144433186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837019fe70 con 0x7f8370102b40 2024-04-23T18:12:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f8374aa4640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f836800cad0 tx=0x7f836800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836800ae20 con 0x7f8370102b40 2024-04-23T18:12:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8368005020 con 0x7f8370102b40 2024-04-23T18:12:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83701a0160 con 0x7f8370102b40 2024-04-23T18:12:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836800f6b0 con 0x7f8370102b40 2024-04-23T18:12:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.996+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83701a06b0 con 0x7f8370102b40 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.997+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83701a0970 con 0x7f8370102b40 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.998+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f83680051d0 con 0x7f8370102b40 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.998+0000 7f836d7fa640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 0x7f83580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.999+0000 7f836ffff640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 0x7f83580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.999+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f83680cffa0 con 0x7f8370102b40 2024-04-23T18:12:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:42.999+0000 7f836ffff640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 0x7f83580a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f83600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:42.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.003+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8368014030 con 0x7f8370102b40 2024-04-23T18:12:43.018 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:42 smithi064 podman[63917]: 2024-04-23 18:12:42.665407061 +0000 UTC m=+0.018740273 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:43.019 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:42 smithi064 podman[63917]: 2024-04-23 18:12:42.772767737 +0000 UTC m=+0.126100930 container create 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:12:43.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.234+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83701a0b40 con 0x7f8370102b40 2024-04-23T18:12:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.235+0000 7f836d7fa640 1 -- 172.21.15.32:0/1144433186 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f836809ce20 con 0x7f8370102b40 2024-04-23T18:12:43.229 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:43.229 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:12:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 6 2024-04-23T18:12:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 msgr2=0x7f83580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 0x7f83580a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f83600314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 msgr2=0x7f837019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f836800cad0 tx=0x7f836800cf90 comp rx=0 tx=0).stop 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 shutdown_connections 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8370102b40 0x7f837019f800 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8370102180 0x7f8370198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f83580a3c30 0x7f83580a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 --2- 172.21.15.32:0/1144433186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8370100f90 0x7f8370198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 >> 172.21.15.32:0/1144433186 conn(0x7f83700fc520 msgr2=0x7f837006a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 shutdown_connections 2024-04-23T18:12:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:43.237+0000 7f8376507640 1 -- 172.21.15.32:0/1144433186 wait complete. 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 podman[63917]: 2024-04-23 18:12:43.025519025 +0000 UTC m=+0.378852218 container init 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 podman[63917]: 2024-04-23 18:12:43.028202494 +0000 UTC m=+0.381535689 container start 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mds[63936]: -- 172.21.15.64:0/3938442441 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556510dc5a40 con 0x556510d98800 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: starting mds.f at 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 bash[63917]: 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a 2024-04-23T18:12:43.297 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:12:43 smithi064 systemd[1]: Started Ceph mds.f for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:43.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:43 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1144433186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:43 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1144433186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:43.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:43.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:43.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:43 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1144433186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:44.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:44 smithi064 ceph-mon[33062]: Deploying daemon mds.c on smithi064 2024-04-23T18:12:44.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:44 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6834/669117639,v1:172.21.15.64:6835/669117639] up:boot 2024-04-23T18:12:44.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:44 smithi064 ceph-mon[33062]: fsmap 6 up:standby 2024-04-23T18:12:44.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:44 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:12:44.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:44 smithi040 ceph-mon[32626]: Deploying daemon mds.c on smithi064 2024-04-23T18:12:44.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:44 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6834/669117639,v1:172.21.15.64:6835/669117639] up:boot 2024-04-23T18:12:44.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:44 smithi040 ceph-mon[32626]: fsmap 6 up:standby 2024-04-23T18:12:44.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:44 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:12:44.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:44 smithi032 ceph-mon[29221]: Deploying daemon mds.c on smithi064 2024-04-23T18:12:44.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:44 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6834/669117639,v1:172.21.15.64:6835/669117639] up:boot 2024-04-23T18:12:44.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:44 smithi032 ceph-mon[29221]: fsmap 6 up:standby 2024-04-23T18:12:44.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:44 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:12:45.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:45.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 -- 172.21.15.32:0/2862119554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414100e60 msgr2=0x7f1414101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 --2- 172.21.15.32:0/2862119554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414100e60 0x7f1414101280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1410009ca0 tx=0x7f141002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 -- 172.21.15.32:0/2862119554 shutdown_connections 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 --2- 172.21.15.32:0/2862119554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 0x7f1414109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 --2- 172.21.15.32:0/2862119554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414102040 0x7f14141024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 --2- 172.21.15.32:0/2862119554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414100e60 0x7f1414101280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 -- 172.21.15.32:0/2862119554 >> 172.21.15.32:0/2862119554 conn(0x7f14140fc420 msgr2=0x7f14140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 -- 172.21.15.32:0/2862119554 shutdown_connections 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.366+0000 7f141c122640 1 -- 172.21.15.32:0/2862119554 wait complete. 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 Processor -- start 2024-04-23T18:12:45.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 -- start start 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414100e60 0x7f141410b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 0x7f141410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f141410db50 con 0x7f1414102040 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f141410d9d0 con 0x7f1414100e60 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141c122640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f141410dcd0 con 0x7f1414102a00 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51438/0 (socket says 172.21.15.32:51438) 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 -- 172.21.15.32:0/916535220 learned_addr learned my addr 172.21.15.32:0/916535220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f141a6bf640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 0x7f141410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f1419ebe640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414100e60 0x7f141410b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 -- 172.21.15.32:0/916535220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 msgr2=0x7f141410a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 0x7f141410a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 -- 172.21.15.32:0/916535220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414100e60 msgr2=0x7f141410b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414100e60 0x7f141410b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.367+0000 7f14196bd640 1 -- 172.21.15.32:0/916535220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141410ac70 con 0x7f1414102040 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f14196bd640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f140400e9f0 tx=0x7f140400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140400cd80 con 0x7f1414102040 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14040108e0 con 0x7f1414102040 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14141a2300 con 0x7f1414102040 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404018980 con 0x7f1414102040 2024-04-23T18:12:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.368+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141a28a0 con 0x7f1414102040 2024-04-23T18:12:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.369+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1414108270 con 0x7f1414102040 2024-04-23T18:12:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.370+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1404010430 con 0x7f1414102040 2024-04-23T18:12:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.371+0000 7f140affd640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 0x7f14000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.371+0000 7f1419ebe640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 0x7f14000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.371+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f14040d0320 con 0x7f1414102040 2024-04-23T18:12:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.371+0000 7f1419ebe640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 0x7f14000a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1410006010 tx=0x7f1410031b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.375+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f140409d100 con 0x7f1414102040 2024-04-23T18:12:45.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:45 smithi064 ceph-mon[33062]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.599+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f141402d5c0 con 0x7f1414102040 2024-04-23T18:12:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.600+0000 7f140affd640 1 -- 172.21.15.32:0/916535220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7f140409ca00 con 0x7f1414102040 2024-04-23T18:12:45.595 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:45.595 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:12:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-04-23T18:12:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.601+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 msgr2=0x7f14000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.601+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 0x7f14000a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1410006010 tx=0x7f1410031b10 comp rx=0 tx=0).stop 2024-04-23T18:12:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.601+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 msgr2=0x7f1414109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.601+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f140400e9f0 tx=0x7f140400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 shutdown_connections 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1414102a00 0x7f141410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1414102040 0x7f1414109ea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f14000a3c30 0x7f14000a60f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 --2- 172.21.15.32:0/916535220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1414100e60 0x7f141410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 >> 172.21.15.32:0/916535220 conn(0x7f14140fc420 msgr2=0x7f1414105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 shutdown_connections 2024-04-23T18:12:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:45.602+0000 7f141c122640 1 -- 172.21.15.32:0/916535220 wait complete. 2024-04-23T18:12:45.605 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:45 smithi032 ceph-mon[29221]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:45.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:45 smithi040 ceph-mon[32626]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:46.053 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:12:45 smithi064 systemd[1]: Started Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:46.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/916535220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:46.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:46.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:46 smithi064 ceph-mon[33062]: Deploying daemon mds.h on smithi040 2024-04-23T18:12:46.573 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/916535220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:46.573 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.573 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.574 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.574 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:46.574 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:46.574 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:46.574 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:46 smithi040 ceph-mon[32626]: Deploying daemon mds.h on smithi040 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/916535220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:46.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:46.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:46 smithi032 ceph-mon[29221]: Deploying daemon mds.h on smithi040 2024-04-23T18:12:47.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:47.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:47 smithi040 ceph-mon[32626]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:47.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:47 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6836/1625538187,v1:172.21.15.64:6837/1625538187] up:boot 2024-04-23T18:12:47.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:47 smithi040 ceph-mon[32626]: fsmap 7 up:standby 2024-04-23T18:12:47.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:47 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:12:47.633 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:47 smithi032 ceph-mon[29221]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:47.633 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:47 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6836/1625538187,v1:172.21.15.64:6837/1625538187] up:boot 2024-04-23T18:12:47.634 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:47 smithi032 ceph-mon[29221]: fsmap 7 up:standby 2024-04-23T18:12:47.634 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:47 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:12:47.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:47 smithi064 ceph-mon[33062]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:47.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:47 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6836/1625538187,v1:172.21.15.64:6837/1625538187] up:boot 2024-04-23T18:12:47.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:47 smithi064 ceph-mon[33062]: fsmap 7 up:standby 2024-04-23T18:12:47.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:47 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 -- 172.21.15.32:0/2242681506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8102a00 msgr2=0x7fc5e8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 --2- 172.21.15.32:0/2242681506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8102a00 0x7fc5e8109320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009ca0 tx=0x7fc5e002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 -- 172.21.15.32:0/2242681506 shutdown_connections 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 --2- 172.21.15.32:0/2242681506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8102a00 0x7fc5e8109320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 --2- 172.21.15.32:0/2242681506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 --2- 172.21.15.32:0/2242681506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8100e60 0x7fc5e8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 -- 172.21.15.32:0/2242681506 >> 172.21.15.32:0/2242681506 conn(0x7fc5e80fc460 msgr2=0x7fc5e80fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 -- 172.21.15.32:0/2242681506 shutdown_connections 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.728+0000 7fc5f0631640 1 -- 172.21.15.32:0/2242681506 wait complete. 2024-04-23T18:12:47.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 Processor -- start 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 -- start start 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 0x7fc5e819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e810ca50 con 0x7fc5e8100e60 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e810c8d0 con 0x7fc5e8102a00 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5f0631640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e810cbd0 con 0x7fc5e8102040 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42106/0 (socket says 172.21.15.32:42106) 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 -- 172.21.15.32:0/1310638906 learned_addr learned my addr 172.21.15.32:0/1310638906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5eebce640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 0x7fc5e819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5edbcc640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:47.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 -- 172.21.15.32:0/1310638906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 msgr2=0x7fc5e819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 -- 172.21.15.32:0/1310638906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 msgr2=0x7fc5e819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 0x7fc5e819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5ee3cd640 1 -- 172.21.15.32:0/1310638906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e819fdb0 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5edbcc640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.729+0000 7fc5eebce640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 0x7fc5e819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5ee3cd640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400d770 tx=0x7fc5e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4017070 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5e4009c30 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e81a00a0 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4003cc0 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.730+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e81a05f0 con 0x7fc5e8100e60 2024-04-23T18:12:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.731+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e81081a0 con 0x7fc5e8100e60 2024-04-23T18:12:47.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.733+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc5e4003e60 con 0x7fc5e8100e60 2024-04-23T18:12:47.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.733+0000 7fc5df7fe640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 0x7fc5d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:47.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.733+0000 7fc5edbcc640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 0x7fc5d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:47.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.733+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5e40cfde0 con 0x7fc5e8100e60 2024-04-23T18:12:47.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.734+0000 7fc5edbcc640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 0x7fc5d40a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5e806afa0 tx=0x7fc5d8005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:47.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.737+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5e4009de0 con 0x7fc5e8100e60 2024-04-23T18:12:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.964+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5e802d5c0 con 0x7fc5e8100e60 2024-04-23T18:12:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.964+0000 7fc5df7fe640 1 -- 172.21.15.32:0/1310638906 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6337 (secure 0 0 0) 0x7fc5e409cc60 con 0x7fc5e8100e60 2024-04-23T18:12:47.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:47.959 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T18:12:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-04-23T18:12:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 msgr2=0x7fc5d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 0x7fc5d40a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5e806afa0 tx=0x7fc5d8005d30 comp rx=0 tx=0).stop 2024-04-23T18:12:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 msgr2=0x7fc5e806b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400d770 tx=0x7fc5e400dc30 comp rx=0 tx=0).stop 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 shutdown_connections 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc5d40a3bb0 0x7fc5d40a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8102a00 0x7fc5e819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e8102040 0x7fc5e819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 --2- 172.21.15.32:0/1310638906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5e8100e60 0x7fc5e806b230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 >> 172.21.15.32:0/1310638906 conn(0x7fc5e80fc460 msgr2=0x7fc5e8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 shutdown_connections 2024-04-23T18:12:47.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:47.966+0000 7fc5f0631640 1 -- 172.21.15.32:0/1310638906 wait complete. 2024-04-23T18:12:48.315 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 systemd[1]: Starting Ceph mds.h for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:48.606 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:48 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1310638906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:48.607 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 podman[62490]: 2024-04-23 18:12:48.392163425 +0000 UTC m=+0.018508575 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:48.607 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 podman[62490]: 2024-04-23 18:12:48.506239519 +0000 UTC m=+0.132584654 container create e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:12:48.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:48 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1310638906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:48.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:48 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1310638906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:49.117 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 podman[62490]: 2024-04-23 18:12:48.759221583 +0000 UTC m=+0.385566721 container init e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T18:12:49.117 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 podman[62490]: 2024-04-23 18:12:48.762240303 +0000 UTC m=+0.388585439 container start e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T18:12:49.117 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 ceph-mds[62508]: -- 172.21.15.40:0/1016051107 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f59a631a40 con 0x55f59a605000 2024-04-23T18:12:49.118 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: starting mds.h at 2024-04-23T18:12:49.118 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 bash[62490]: e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 2024-04-23T18:12:49.118 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:12:48 smithi040 systemd[1]: Started Ceph mds.h for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:49.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 -- 172.21.15.32:0/3817260427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 msgr2=0x7fdc74101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3817260427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74101280 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009900 tx=0x7fdc5c02a410 comp rx=0 tx=0).stop 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 -- 172.21.15.32:0/3817260427 shutdown_connections 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3817260427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102a00 0x7fdc74109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3817260427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102040 0x7fdc741024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3817260427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74101280 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 -- 172.21.15.32:0/3817260427 >> 172.21.15.32:0/3817260427 conn(0x7fdc740fc420 msgr2=0x7fdc740fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 -- 172.21.15.32:0/3817260427 shutdown_connections 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 -- 172.21.15.32:0/3817260427 wait complete. 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.093+0000 7fdc797a2640 1 Processor -- start 2024-04-23T18:12:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 -- start start 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102040 0x7fdc741034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102a00 0x7fdc741039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7410c9d0 con 0x7fdc74100e60 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7410c850 con 0x7fdc74102040 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc797a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7410cb50 con 0x7fdc74102a00 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42134/0 (socket says 172.21.15.32:42134) 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 -- 172.21.15.32:0/3246147638 learned_addr learned my addr 172.21.15.32:0/3246147638 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc727fc640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102040 0x7fdc741034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc737fe640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102a00 0x7fdc741039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 -- 172.21.15.32:0/3246147638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102a00 msgr2=0x7fdc741039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102a00 0x7fdc741039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 -- 172.21.15.32:0/3246147638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102040 msgr2=0x7fdc741034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102040 0x7fdc741034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 -- 172.21.15.32:0/3246147638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc741042b0 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.094+0000 7fdc72ffd640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006010 tx=0x7fdc5c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.095+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c03f030 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.095+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc741a2310 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.095+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5c003cb0 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.095+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c035070 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.095+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc741a2820 con 0x7fdc74100e60 2024-04-23T18:12:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.096+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc741a2a80 con 0x7fdc74100e60 2024-04-23T18:12:50.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.097+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdc5c042030 con 0x7fdc74100e60 2024-04-23T18:12:50.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.097+0000 7fdc6bfff640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 0x7fdc600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:50.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.098+0000 7fdc727fc640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 0x7fdc600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:50.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.098+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdc5c0f06d0 con 0x7fdc74100e60 2024-04-23T18:12:50.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.098+0000 7fdc727fc640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 0x7fdc600a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdc64005fd0 tx=0x7fdc64005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:50.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.102+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc5c0bd4b0 con 0x7fdc74100e60 2024-04-23T18:12:50.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:50.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:boot 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: fsmap 8 up:standby 2024-04-23T18:12:50.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:49 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:boot 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: fsmap 8 up:standby 2024-04-23T18:12:50.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:49 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:12:50.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.329+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc741a2c50 con 0x7fdc74100e60 2024-04-23T18:12:50.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.329+0000 7fdc6bfff640 1 -- 172.21.15.32:0/3246147638 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7162 (secure 0 0 0) 0x7fdc5c0bcdb0 con 0x7fdc74100e60 2024-04-23T18:12:50.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:50.324 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:12:50.324 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 9 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 msgr2=0x7fdc600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 0x7fdc600a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdc64005fd0 tx=0x7fdc64005850 comp rx=0 tx=0).stop 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 msgr2=0x7fdc74104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006010 tx=0x7fdc5c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 shutdown_connections 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc74102a00 0x7fdc741039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdc600a3bb0 0x7fdc600a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc74102040 0x7fdc741034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 --2- 172.21.15.32:0/3246147638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74100e60 0x7fdc74104e20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.331+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 >> 172.21.15.32:0/3246147638 conn(0x7fdc740fc420 msgr2=0x7fdc740fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.332+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 shutdown_connections 2024-04-23T18:12:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:50.332+0000 7fdc797a2640 1 -- 172.21.15.32:0/3246147638 wait complete. 2024-04-23T18:12:50.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:50.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:50.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:boot 2024-04-23T18:12:50.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: fsmap 8 up:standby 2024-04-23T18:12:50.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:49 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:12:51.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:50 smithi064 ceph-mon[33062]: Deploying daemon mds.i on smithi064 2024-04-23T18:12:51.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:50 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3246147638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:51.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:50 smithi032 ceph-mon[29221]: Deploying daemon mds.i on smithi064 2024-04-23T18:12:51.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:50 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3246147638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:51.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:50 smithi040 ceph-mon[32626]: Deploying daemon mds.i on smithi064 2024-04-23T18:12:51.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:50 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3246147638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:51.422 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 systemd[1]: Starting Ceph mds.i for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:51.738 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 podman[64538]: 2024-04-23 18:12:51.38343585 +0000 UTC m=+0.018732056 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:51.739 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 podman[64538]: 2024-04-23 18:12:51.490971837 +0000 UTC m=+0.126268028 container create 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:12:51.739 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 podman[64538]: 2024-04-23 18:12:51.718521293 +0000 UTC m=+0.353817487 container init 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:12:51.739 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 podman[64538]: 2024-04-23 18:12:51.721796043 +0000 UTC m=+0.357092241 container start 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:12:51.739 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mds[64557]: -- 172.21.15.64:0/1560770694 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e43d87da40 con 0x55e43d850800 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: starting mds.i at 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 bash[64538]: 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:12:51 smithi064 systemd[1]: Started Ceph mds.i for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:52.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:51 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:52.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:51 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:51 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 -- 172.21.15.32:0/4137258063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294102b70 msgr2=0x7fa294104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 --2- 172.21.15.32:0/4137258063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294102b70 0x7fa294104f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa284009900 tx=0x7fa28402a410 comp rx=0 tx=0).stop 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 -- 172.21.15.32:0/4137258063 shutdown_connections 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 --2- 172.21.15.32:0/4137258063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294107dd0 0x7fa29410a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 --2- 172.21.15.32:0/4137258063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 0x7fa294107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 --2- 172.21.15.32:0/4137258063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294102b70 0x7fa294104f60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.457+0000 7fa2993d8640 1 -- 172.21.15.32:0/4137258063 >> 172.21.15.32:0/4137258063 conn(0x7fa2940fc520 msgr2=0x7fa2940fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.458+0000 7fa2993d8640 1 -- 172.21.15.32:0/4137258063 shutdown_connections 2024-04-23T18:12:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.458+0000 7fa2993d8640 1 -- 172.21.15.32:0/4137258063 wait complete. 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.458+0000 7fa2993d8640 1 Processor -- start 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 -- start start 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294102b70 0x7fa29419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 0x7fa29419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa29410db50 con 0x7fa294107dd0 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa29410d9d0 con 0x7fa2941054a0 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2993d8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa29410dcd0 con 0x7fa294102b70 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42158/0 (socket says 172.21.15.32:42158) 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 -- 172.21.15.32:0/3596375381 learned_addr learned my addr 172.21.15.32:0/3596375381 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:52.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa28a7fc640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 0x7fa29419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa292ffd640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294102b70 0x7fa29419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 -- 172.21.15.32:0/3596375381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294102b70 msgr2=0x7fa29419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294102b70 0x7fa29419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 -- 172.21.15.32:0/3596375381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 msgr2=0x7fa29419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 0x7fa29419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 -- 172.21.15.32:0/3596375381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2941a1fe0 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.459+0000 7fa2937fe640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b970 tx=0x7fa28c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.460+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c019070 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.460+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2941a22d0 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.460+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa28c014440 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.460+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c0075d0 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.460+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2941a28f0 con 0x7fa294107dd0 2024-04-23T18:12:52.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.461+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2941098c0 con 0x7fa294107dd0 2024-04-23T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.462+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa28c01c070 con 0x7fa294107dd0 2024-04-23T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.462+0000 7fa290ff9640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 0x7fa2800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.463+0000 7fa292ffd640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 0x7fa2800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.463+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa28c0cf640 con 0x7fa294107dd0 2024-04-23T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.463+0000 7fa292ffd640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 0x7fa2800a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa284000c00 tx=0x7fa2840314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:52.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.467+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa28c09c420 con 0x7fa294107dd0 2024-04-23T18:12:52.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.717+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa294062b50 con 0x7fa294107dd0 2024-04-23T18:12:52.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.718+0000 7fa290ff9640 1 -- 172.21.15.32:0/3596375381 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7989 (secure 0 0 0) 0x7fa28c09bd20 con 0x7fa294107dd0 2024-04-23T18:12:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:52.713 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:12:52.713 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 10 2024-04-23T18:12:52.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 msgr2=0x7fa2800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 0x7fa2800a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa284000c00 tx=0x7fa2840314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 msgr2=0x7fa2941a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b970 tx=0x7fa28c00be30 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 shutdown_connections 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa294107dd0 0x7fa2941a1970 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa2800a3bb0 0x7fa2800a6070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2941054a0 0x7fa29419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 --2- 172.21.15.32:0/3596375381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa294102b70 0x7fa29419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 >> 172.21.15.32:0/3596375381 conn(0x7fa2940fc520 msgr2=0x7fa294102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 shutdown_connections 2024-04-23T18:12:52.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:52.720+0000 7fa2993d8640 1 -- 172.21.15.32:0/3596375381 wait complete. 2024-04-23T18:12:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:52 smithi032 ceph-mon[29221]: Deploying daemon mds.g on smithi032 2024-04-23T18:12:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:52 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:boot 2024-04-23T18:12:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:52 smithi032 ceph-mon[29221]: fsmap 9 up:standby 2024-04-23T18:12:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:52 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:12:53.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:52 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3596375381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:53.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:52 smithi064 ceph-mon[33062]: Deploying daemon mds.g on smithi032 2024-04-23T18:12:53.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:52 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:boot 2024-04-23T18:12:53.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:52 smithi064 ceph-mon[33062]: fsmap 9 up:standby 2024-04-23T18:12:53.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:52 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:12:53.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:52 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3596375381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:53.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:52 smithi040 ceph-mon[32626]: Deploying daemon mds.g on smithi032 2024-04-23T18:12:53.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:52 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:boot 2024-04-23T18:12:53.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:52 smithi040 ceph-mon[32626]: fsmap 9 up:standby 2024-04-23T18:12:53.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:52 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:12:53.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:52 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3596375381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:54.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:53 smithi064 ceph-mon[33062]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:54.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:53 smithi032 ceph-mon[29221]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:54.258 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 systemd[1]: Starting Ceph mds.g for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:54.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:53 smithi040 ceph-mon[32626]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:54.624 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 podman[70917]: 2024-04-23 18:12:54.263388451 +0000 UTC m=+0.018118379 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:54.624 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 podman[70917]: 2024-04-23 18:12:54.386561555 +0000 UTC m=+0.141291460 container create 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T18:12:54.624 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 podman[70917]: 2024-04-23 18:12:54.630868002 +0000 UTC m=+0.385597908 container init 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:12:54.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 -- 172.21.15.32:0/2592138944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4107c90 msgr2=0x7efcd410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 --2- 172.21.15.32:0/2592138944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4107c90 0x7efcd410a0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efccc009ca0 tx=0x7efccc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 -- 172.21.15.32:0/2592138944 shutdown_connections 2024-04-23T18:12:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 --2- 172.21.15.32:0/2592138944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4107c90 0x7efcd410a0e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 --2- 172.21.15.32:0/2592138944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4105360 0x7efcd4107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 --2- 172.21.15.32:0/2592138944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4102a30 0x7efcd4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 -- 172.21.15.32:0/2592138944 >> 172.21.15.32:0/2592138944 conn(0x7efcd40fc460 msgr2=0x7efcd40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 -- 172.21.15.32:0/2592138944 shutdown_connections 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.848+0000 7efcdb737640 1 -- 172.21.15.32:0/2592138944 wait complete. 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 Processor -- start 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 -- start start 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 0x7efcd41009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 0x7efcd4100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410db40 con 0x7efcd4102a30 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410d9c0 con 0x7efcd4107c90 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcdb737640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410dcc0 con 0x7efcd4105360 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42204/0 (socket says 172.21.15.32:42204) 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 -- 172.21.15.32:0/3626234942 learned_addr learned my addr 172.21.15.32:0/3626234942 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd9cd4640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 0x7efcd4100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd8cd2640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 0x7efcd41009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 -- 172.21.15.32:0/3626234942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 msgr2=0x7efcd41009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 0x7efcd41009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 -- 172.21.15.32:0/3626234942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 msgr2=0x7efcd4100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 0x7efcd4100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 -- 172.21.15.32:0/3626234942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd41017b0 con 0x7efcd4102a30 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd8cd2640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 0x7efcd41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd9cd4640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 0x7efcd4100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.849+0000 7efcd94d3640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efcd000d950 tx=0x7efcd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.850+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0017030 con 0x7efcd4102a30 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.850+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0004020 con 0x7efcd4102a30 2024-04-23T18:12:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.850+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd41a2290 con 0x7efcd4102a30 2024-04-23T18:12:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.850+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd00047a0 con 0x7efcd4102a30 2024-04-23T18:12:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.850+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd41a27e0 con 0x7efcd4102a30 2024-04-23T18:12:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.851+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd40fdf50 con 0x7efcd4102a30 2024-04-23T18:12:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.852+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efcd0009a30 con 0x7efcd4102a30 2024-04-23T18:12:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.853+0000 7efcca7fc640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 0x7efcc00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.853+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efcd00cfb20 con 0x7efcd4102a30 2024-04-23T18:12:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.853+0000 7efcd8cd2640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 0x7efcc00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.853+0000 7efcd8cd2640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 0x7efcc00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efcd4102110 tx=0x7efcc40094b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:54.857+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efcd009c9a0 con 0x7efcd4102a30 2024-04-23T18:12:55.008 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 podman[70917]: 2024-04-23 18:12:54.633617949 +0000 UTC m=+0.388347855 container start 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:12:55.008 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 ceph-mds[70936]: -- 172.21.15.32:0/2406883349 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561dc818da40 con 0x561dc8161000 2024-04-23T18:12:55.008 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g[70932]: starting mds.g at 2024-04-23T18:12:55.008 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 bash[70917]: 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 2024-04-23T18:12:55.008 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:12:54 smithi032 systemd[1]: Started Ceph mds.g for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.087+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcd402d5c0 con 0x7efcd4102a30 2024-04-23T18:12:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.088+0000 7efcca7fc640 1 -- 172.21.15.32:0/3626234942 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7efcd009c2a0 con 0x7efcd4102a30 2024-04-23T18:12:55.082 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:55.082 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:12:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 11 2024-04-23T18:12:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 msgr2=0x7efcc00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 0x7efcc00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efcd4102110 tx=0x7efcc40094b0 comp rx=0 tx=0).stop 2024-04-23T18:12:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 msgr2=0x7efcd41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efcd000d950 tx=0x7efcd000de10 comp rx=0 tx=0).stop 2024-04-23T18:12:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 shutdown_connections 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcc00a3bb0 0x7efcc00a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcd4107c90 0x7efcd4100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4105360 0x7efcd41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 --2- 172.21.15.32:0/3626234942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd4102a30 0x7efcd41023a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 >> 172.21.15.32:0/3626234942 conn(0x7efcd40fc460 msgr2=0x7efcd4103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 shutdown_connections 2024-04-23T18:12:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:55.090+0000 7efcdb737640 1 -- 172.21.15.32:0/3626234942 wait complete. 2024-04-23T18:12:56.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:56.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: Deploying daemon mds.a on smithi032 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.32:6838/156606909,v1:172.21.15.32:6839/156606909] up:boot 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: fsmap 10 up:standby 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:12:56.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:55 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3626234942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: Deploying daemon mds.a on smithi032 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.32:6838/156606909,v1:172.21.15.32:6839/156606909] up:boot 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: fsmap 10 up:standby 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:12:56.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:55 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3626234942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:56.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: Deploying daemon mds.a on smithi032 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.32:6838/156606909,v1:172.21.15.32:6839/156606909] up:boot 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: fsmap 10 up:standby 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:12:56.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:55 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3626234942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:57.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:57.203 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 systemd[1]: Starting Ceph mds.a for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- 172.21.15.32:0/919268764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8100830 msgr2=0x7f4fb8100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/919268764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8100830 0x7f4fb8100c50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009900 tx=0x7f4fa802a410 comp rx=0 tx=0).stop 2024-04-23T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- 172.21.15.32:0/919268764 shutdown_connections 2024-04-23T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/919268764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101b30 0x7f4fb8101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/919268764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8101190 0x7f4fb81015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/919268764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8100830 0x7f4fb8100c50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- 172.21.15.32:0/919268764 >> 172.21.15.32:0/919268764 conn(0x7f4fb80fc430 msgr2=0x7f4fb80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- 172.21.15.32:0/919268764 shutdown_connections 2024-04-23T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- 172.21.15.32:0/919268764 wait complete. 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 Processor -- start 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.241+0000 7f4fbe39b640 1 -- start start 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8100830 0x7f4fb819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101190 0x7f4fb819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb810fd10 con 0x7f4fb8101b30 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb810fb90 con 0x7f4fb8100830 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbe39b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb810fe90 con 0x7f4fb8101190 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48572/0 (socket says 172.21.15.32:48572) 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 -- 172.21.15.32:0/2827833755 learned_addr learned my addr 172.21.15.32:0/2827833755 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fb7fff640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8100830 0x7f4fb819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fb77fe640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101190 0x7f4fb819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 -- 172.21.15.32:0/2827833755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101190 msgr2=0x7f4fb819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101190 0x7f4fb819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.242+0000 7f4fbc938640 1 -- 172.21.15.32:0/2827833755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8100830 msgr2=0x7f4fb819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fbc938640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8100830 0x7f4fb819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fbc938640 1 -- 172.21.15.32:0/2827833755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb81a1fc0 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fbc938640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000eb10 tx=0x7f4fb000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0004b20 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fb0004cc0 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0010670 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb81a22b0 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.243+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb81a2850 con 0x7f4fb8101b30 2024-04-23T18:12:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.244+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb810b710 con 0x7f4fb8101b30 2024-04-23T18:12:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.246+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4fb0005000 con 0x7f4fb8101b30 2024-04-23T18:12:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.246+0000 7f4fb57fa640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 0x7f4fa00a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.246+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4fb00d06d0 con 0x7f4fb8101b30 2024-04-23T18:12:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.246+0000 7f4fb7fff640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 0x7f4fa00a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.247+0000 7f4fb7fff640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 0x7f4fa00a6170 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8000c00 tx=0x7f4fa80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.254+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fb0053ff0 con 0x7f4fb8101b30 2024-04-23T18:12:57.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.503+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fb8062b50 con 0x7f4fb8101b30 2024-04-23T18:12:57.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.504+0000 7f4fb57fa640 1 -- 172.21.15.32:0/2827833755 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7f4fb009d4b0 con 0x7f4fb8101b30 2024-04-23T18:12:57.498 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:57.498 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:12:57.498 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 11 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 msgr2=0x7f4fa00a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 0x7f4fa00a6170 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8000c00 tx=0x7f4fa80314a0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 msgr2=0x7f4fb81a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000eb10 tx=0x7f4fb000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 shutdown_connections 2024-04-23T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb8101b30 0x7f4fb81a1950 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb8101190 0x7f4fb819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4fa00a3cb0 0x7f4fa00a6170 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 --2- 172.21.15.32:0/2827833755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb8100830 0x7f4fb819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 >> 172.21.15.32:0/2827833755 conn(0x7f4fb80fc430 msgr2=0x7f4fb80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 shutdown_connections 2024-04-23T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:57.506+0000 7f4fbe39b640 1 -- 172.21.15.32:0/2827833755 wait complete. 2024-04-23T18:12:57.507 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 podman[71311]: 2024-04-23 18:12:57.315611643 +0000 UTC m=+0.026335726 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:12:57.507 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 podman[71311]: 2024-04-23 18:12:57.432872347 +0000 UTC m=+0.143596417 container create 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:12:57.794 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 podman[71311]: 2024-04-23 18:12:57.71030601 +0000 UTC m=+0.421030081 container init 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:12:57.794 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 podman[71311]: 2024-04-23 18:12:57.712988543 +0000 UTC m=+0.423712629 container start 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:12:57.794 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 ceph-mds[71333]: -- 172.21.15.32:0/3798843130 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56252d99da40 con 0x56252d970c00 2024-04-23T18:12:57.794 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a[71329]: starting mds.a at 2024-04-23T18:12:58.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:57 smithi040 ceph-mon[32626]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:58.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2827833755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:58.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:57 smithi064 ceph-mon[33062]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:58.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2827833755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:58.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:57 smithi032 ceph-mon[29221]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:12:58.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2827833755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:12:58.258 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 bash[71311]: 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 2024-04-23T18:12:58.258 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:12:57 smithi032 systemd[1]: Started Ceph mds.a for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.32:6840/2029849926,v1:172.21.15.32:6841/2029849926] up:boot 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: fsmap 11 up:standby 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:58.823 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:58 smithi040 ceph-mon[32626]: Deploying daemon mds.b on smithi040 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.32:6840/2029849926,v1:172.21.15.32:6841/2029849926] up:boot 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: fsmap 11 up:standby 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:59.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:59.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:59.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:58 smithi064 ceph-mon[33062]: Deploying daemon mds.b on smithi040 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.32:6840/2029849926,v1:172.21.15.32:6841/2029849926] up:boot 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: fsmap 11 up:standby 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:59.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:58 smithi032 ceph-mon[29221]: Deploying daemon mds.b on smithi040 2024-04-23T18:12:59.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:12:59.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 -- 172.21.15.32:0/1899231830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 msgr2=0x7f8b50109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/1899231830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b50109320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009ca0 tx=0x7f8b4802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 -- 172.21.15.32:0/1899231830 shutdown_connections 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/1899231830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b50109320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/1899231830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50102040 0x7f8b501024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/1899231830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50100e60 0x7f8b50101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 -- 172.21.15.32:0/1899231830 >> 172.21.15.32:0/1899231830 conn(0x7f8b500fc420 msgr2=0x7f8b500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 -- 172.21.15.32:0/1899231830 shutdown_connections 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.636+0000 7f8b57ce1640 1 -- 172.21.15.32:0/1899231830 wait complete. 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 Processor -- start 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 -- start start 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 0x7f8b50198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 0x7f8b50198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010caf0 con 0x7f8b50102a00 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010c970 con 0x7f8b50102040 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b57ce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010cc70 con 0x7f8b50100e60 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48612/0 (socket says 172.21.15.32:48612) 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 -- 172.21.15.32:0/3212920600 learned_addr learned my addr 172.21.15.32:0/3212920600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5527c640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 0x7f8b50198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b55a7d640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 0x7f8b50198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 -- 172.21.15.32:0/3212920600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 msgr2=0x7f8b50198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 0x7f8b50198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 -- 172.21.15.32:0/3212920600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 msgr2=0x7f8b50198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 0x7f8b50198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 -- 172.21.15.32:0/3212920600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5019fee0 con 0x7f8b50102a00 2024-04-23T18:12:59.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5527c640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 0x7f8b50198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b55a7d640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 0x7f8b50198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.637+0000 7f8b5627e640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b48031a20 tx=0x7f8b48003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.638+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48004240 con 0x7f8b50102a00 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.638+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b48003e10 con 0x7f8b50102a00 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.638+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480305c0 con 0x7f8b50102a00 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.638+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b501a0170 con 0x7f8b50102a00 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.638+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b501a0790 con 0x7f8b50102a00 2024-04-23T18:12:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.639+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b501a09f0 con 0x7f8b50102a00 2024-04-23T18:12:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.640+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8b4802f660 con 0x7f8b50102a00 2024-04-23T18:12:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.640+0000 7f8b46ffd640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 0x7f8b3c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.641+0000 7f8b55a7d640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 0x7f8b3c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.641+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8b480f19b0 con 0x7f8b50102a00 2024-04-23T18:12:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.641+0000 7f8b55a7d640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 0x7f8b3c0a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c006fd0 tx=0x7f8b4c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:59.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.644+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b480393b0 con 0x7f8b50102a00 2024-04-23T18:12:59.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.871+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b501a0bc0 con 0x7f8b50102a00 2024-04-23T18:12:59.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.871+0000 7f8b46ffd640 1 -- 172.21.15.32:0/3212920600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9638 (secure 0 0 0) 0x7f8b480be8c0 con 0x7f8b50102a00 2024-04-23T18:12:59.866 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:12:59.866 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:12:59.866 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 12 2024-04-23T18:12:59.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 msgr2=0x7f8b3c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 0x7f8b3c0a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c006fd0 tx=0x7f8b4c008040 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 msgr2=0x7f8b5019f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b48031a20 tx=0x7f8b48003c20 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 shutdown_connections 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b50102a00 0x7f8b5019f870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8b3c0a3c30 0x7f8b3c0a60f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b50102040 0x7f8b50198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 --2- 172.21.15.32:0/3212920600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b50100e60 0x7f8b50198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 >> 172.21.15.32:0/3212920600 conn(0x7f8b500fc420 msgr2=0x7f8b500fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 shutdown_connections 2024-04-23T18:12:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:12:59.873+0000 7f8b57ce1640 1 -- 172.21.15.32:0/3212920600 wait complete. 2024-04-23T18:13:00.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:59 smithi064 ceph-mon[33062]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:00.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:12:59 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3212920600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:59 smithi032 ceph-mon[29221]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:00.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:12:59 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3212920600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:00.330 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:59 smithi040 ceph-mon[32626]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:00.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:12:59 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3212920600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:00.617 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 podman[62795]: 2024-04-23 18:13:00.57077459 +0000 UTC m=+0.127565216 container create d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:13:01.117 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 podman[62795]: 2024-04-23 18:13:00.823808815 +0000 UTC m=+0.380599444 container init d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:13:01.117 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 podman[62795]: 2024-04-23 18:13:00.826636069 +0000 UTC m=+0.383426702 container start d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:13:01.117 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 ceph-mds[62814]: -- 172.21.15.40:0/606024884 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c49007a40 con 0x560c48fdac00 2024-04-23T18:13:01.118 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: starting mds.b at 2024-04-23T18:13:01.118 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 bash[62795]: d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 2024-04-23T18:13:01.118 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:00 smithi040 systemd[1]: Started Ceph mds.b for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:13:01.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:13:01.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.988+0000 7f163e4c2640 1 -- 172.21.15.32:0/245092238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 msgr2=0x7f163810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:01.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.988+0000 7f163e4c2640 1 --2- 172.21.15.32:0/245092238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163810a0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1630009ca0 tx=0x7f163002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.988+0000 7f163e4c2640 1 -- 172.21.15.32:0/245092238 shutdown_connections 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.988+0000 7f163e4c2640 1 --2- 172.21.15.32:0/245092238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163810a0e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.988+0000 7f163e4c2640 1 --2- 172.21.15.32:0/245092238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638105360 0x7f1638107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 --2- 172.21.15.32:0/245092238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638102a30 0x7f1638104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- 172.21.15.32:0/245092238 >> 172.21.15.32:0/245092238 conn(0x7f16380fc420 msgr2=0x7f16380fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- 172.21.15.32:0/245092238 shutdown_connections 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- 172.21.15.32:0/245092238 wait complete. 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 Processor -- start 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- start start 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 0x7f16381981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 0x7f1638198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f163810dbc0 con 0x7f1638107c90 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f163810da40 con 0x7f1638102a30 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.989+0000 7f163e4c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f163810dd40 con 0x7f1638105360 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48640/0 (socket says 172.21.15.32:48640) 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 -- 172.21.15.32:0/4140047476 learned_addr learned my addr 172.21.15.32:0/4140047476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f16377fe640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 0x7f1638198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f1637fff640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 0x7f16381981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 -- 172.21.15.32:0/4140047476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 msgr2=0x7f1638198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 0x7f1638198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 -- 172.21.15.32:0/4140047476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 msgr2=0x7f16381981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 0x7f16381981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 -- 172.21.15.32:0/4140047476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163819fe00 con 0x7f1638107c90 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f1637fff640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 0x7f16381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f16377fe640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 0x7f1638198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163ca5f640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1630033040 tx=0x7f16300317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630003b70 con 0x7f1638107c90 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1630003d10 con 0x7f1638107c90 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16381a0090 con 0x7f1638107c90 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163002fb10 con 0x7f1638107c90 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.990+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16381a0660 con 0x7f1638107c90 2024-04-23T18:13:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.991+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16380fdf10 con 0x7f1638107c90 2024-04-23T18:13:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.992+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1630004610 con 0x7f1638107c90 2024-04-23T18:13:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.993+0000 7f16357fa640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 0x7f16240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.993+0000 7f1637fff640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 0x7f16240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.993+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f16300f14c0 con 0x7f1638107c90 2024-04-23T18:13:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.994+0000 7f1637fff640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 0x7f16240a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1620009b50 tx=0x7f1620005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:01.998+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16300be340 con 0x7f1638107c90 2024-04-23T18:13:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:boot 2024-04-23T18:13:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: fsmap 12 up:standby 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:01 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:boot 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: fsmap 12 up:standby 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:01 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:boot 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: fsmap 12 up:standby 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:01 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.228+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1638062b50 con 0x7f1638107c90 2024-04-23T18:13:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.228+0000 7f16357fa640 1 -- 172.21.15.32:0/4140047476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10464 (secure 0 0 0) 0x7f16300bdc40 con 0x7f1638107c90 2024-04-23T18:13:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:02.223 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-23T18:13:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.230+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 msgr2=0x7f16240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.230+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 0x7f16240a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1620009b50 tx=0x7f1620005f90 comp rx=0 tx=0).stop 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.230+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 msgr2=0x7f163819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.230+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1630033040 tx=0x7f16300317c0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 shutdown_connections 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1638107c90 0x7f163819f790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1638105360 0x7f1638198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f16240a3bb0 0x7f16240a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 --2- 172.21.15.32:0/4140047476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1638102a30 0x7f16381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 >> 172.21.15.32:0/4140047476 conn(0x7f16380fc420 msgr2=0x7f1638103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 shutdown_connections 2024-04-23T18:13:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.231+0000 7f163e4c2640 1 -- 172.21.15.32:0/4140047476 wait complete. 2024-04-23T18:13:02.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:13:02.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 -- 172.21.15.32:0/2568653861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4100e60 msgr2=0x7f68a4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2568653861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4100e60 0x7f68a4101280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6894009900 tx=0x7f689402a410 comp rx=0 tx=0).stop 2024-04-23T18:13:02.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 -- 172.21.15.32:0/2568653861 shutdown_connections 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2568653861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4102a00 0x7f68a4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2568653861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102040 0x7f68a41024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2568653861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4100e60 0x7f68a4101280 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 -- 172.21.15.32:0/2568653861 >> 172.21.15.32:0/2568653861 conn(0x7f68a40fc460 msgr2=0x7f68a40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 -- 172.21.15.32:0/2568653861 shutdown_connections 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 -- 172.21.15.32:0/2568653861 wait complete. 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.346+0000 7f68aa44e640 1 Processor -- start 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 -- start start 2024-04-23T18:13:02.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4100e60 0x7f68a4198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 0x7f68a419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68a410db30 con 0x7f68a4102040 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68a410d9b0 con 0x7f68a4100e60 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68aa44e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f68a410dcb0 con 0x7f68a4102a00 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48670/0 (socket says 172.21.15.32:48670) 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 -- 172.21.15.32:0/2107980969 learned_addr learned my addr 172.21.15.32:0/2107980969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a3fff640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4100e60 0x7f68a4198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a89eb640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 0x7f68a419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 -- 172.21.15.32:0/2107980969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 msgr2=0x7f68a419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 0x7f68a419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 -- 172.21.15.32:0/2107980969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4100e60 msgr2=0x7f68a4198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4100e60 0x7f68a4198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 -- 172.21.15.32:0/2107980969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a419fe60 con 0x7f68a4102040 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a89eb640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 0x7f68a419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.347+0000 7f68a37fe640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9f0 tx=0x7f689800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.348+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689800cd80 con 0x7f68a4102040 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.348+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a41a0150 con 0x7f68a4102040 2024-04-23T18:13:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.348+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68980054d0 con 0x7f68a4102040 2024-04-23T18:13:02.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.348+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898010690 con 0x7f68a4102040 2024-04-23T18:13:02.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.348+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a41a0650 con 0x7f68a4102040 2024-04-23T18:13:02.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.349+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a4108270 con 0x7f68a4102040 2024-04-23T18:13:02.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.350+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6898004e90 con 0x7f68a4102040 2024-04-23T18:13:02.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.350+0000 7f68a17fa640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 0x7f688c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.350+0000 7f68a3fff640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 0x7f688c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.350+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f68980d0400 con 0x7f68a4102040 2024-04-23T18:13:02.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.351+0000 7f68a3fff640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 0x7f688c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6894009230 tx=0x7f68940314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.355+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f689809d280 con 0x7f68a4102040 2024-04-23T18:13:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.574+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68a402d7e0 con 0x7f68a4102040 2024-04-23T18:13:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.575+0000 7f68a17fa640 1 -- 172.21.15.32:0/2107980969 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18385 (secure 0 0 0) 0x7f689809cb80 con 0x7f68a4102040 2024-04-23T18:13:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:02.569 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:12:15.803236+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 msgr2=0x7f688c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 0x7f688c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6894009230 tx=0x7f68940314a0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 msgr2=0x7f68a4198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9f0 tx=0x7f689800edf0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 shutdown_connections 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68a4102a00 0x7f68a419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68a4102040 0x7f68a4198780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f688c0a3bb0 0x7f688c0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 --2- 172.21.15.32:0/2107980969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68a4100e60 0x7f68a4198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 >> 172.21.15.32:0/2107980969 conn(0x7f68a40fc460 msgr2=0x7f68a4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 shutdown_connections 2024-04-23T18:13:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.577+0000 7f68aa44e640 1 -- 172.21.15.32:0/2107980969 wait complete. 2024-04-23T18:13:02.584 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:13:02.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:13:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/2583899286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 msgr2=0x7fbe34104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/2583899286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34104e00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009ca0 tx=0x7fbe2402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/2583899286 shutdown_connections 2024-04-23T18:13:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/2583899286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 0x7fbe3410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/2583899286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/2583899286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34104e00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/2583899286 >> 172.21.15.32:0/2583899286 conn(0x7fbe340fc420 msgr2=0x7fbe340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/2583899286 shutdown_connections 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.702+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/2583899286 wait complete. 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 Processor -- start 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 -- start start 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 0x7fbe3419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410db10 con 0x7fbe34102a10 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410d990 con 0x7fbe34107c70 2024-04-23T18:13:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3c6bb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410dc90 con 0x7fbe34105340 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3a457640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3a457640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48674/0 (socket says 172.21.15.32:48674) 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3a457640 1 -- 172.21.15.32:0/4212274339 learned_addr learned my addr 172.21.15.32:0/4212274339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe3ac58640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 0x7fbe3419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 -- 172.21.15.32:0/4212274339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 msgr2=0x7fbe3419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 0x7fbe3419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 -- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 msgr2=0x7fbe34198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.703+0000 7fbe39c56640 1 -- 172.21.15.32:0/4212274339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3419fe40 con 0x7fbe34105340 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe39c56640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34198760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800e9c0 tx=0x7fbe2800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800ccc0 con 0x7fbe34105340 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe341a0130 con 0x7fbe34105340 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe280108e0 con 0x7fbe34105340 2024-04-23T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280187a0 con 0x7fbe34105340 2024-04-23T18:13:02.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.704+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe341a06d0 con 0x7fbe34105340 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.706+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe340fdf10 con 0x7fbe34105340 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.706+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbe28010430 con 0x7fbe34105340 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.707+0000 7fbe337fe640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.707+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbe280d02d0 con 0x7fbe34105340 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.707+0000 7fbe3a457640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.707+0000 7fbe3a457640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005fe0 tx=0x7fbe24005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.712+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe2809ca60 con 0x7fbe34105340 2024-04-23T18:13:02.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.909+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe34062b50 con 0x7fbe34105340 2024-04-23T18:13:02.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.910+0000 7fbe337fe640 1 -- 172.21.15.32:0/4212274339 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fbe280a0d60 con 0x7fbe34105340 2024-04-23T18:13:02.904 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:13:02.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 msgr2=0x7fbe1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005fe0 tx=0x7fbe24005f70 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 msgr2=0x7fbe34198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34198760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800e9c0 tx=0x7fbe2800ee80 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 shutdown_connections 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe34107c70 0x7fbe3419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe34105340 0x7fbe34198760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 --2- 172.21.15.32:0/4212274339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe34102a10 0x7fbe34198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 >> 172.21.15.32:0/4212274339 conn(0x7fbe340fc420 msgr2=0x7fbe341035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 shutdown_connections 2024-04-23T18:13:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:02.912+0000 7fbe3c6bb640 1 -- 172.21.15.32:0/4212274339 wait complete. 2024-04-23T18:13:02.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 -- 172.21.15.32:0/2803855148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 msgr2=0x7fbae010a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 --2- 172.21.15.32:0/2803855148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae010a400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009ca0 tx=0x7fbad802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 -- 172.21.15.32:0/2803855148 shutdown_connections 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 --2- 172.21.15.32:0/2803855148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae010a400 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 --2- 172.21.15.32:0/2803855148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 --2- 172.21.15.32:0/2803855148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 -- 172.21.15.32:0/2803855148 >> 172.21.15.32:0/2803855148 conn(0x7fbae00fd530 msgr2=0x7fbae00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 -- 172.21.15.32:0/2803855148 shutdown_connections 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.033+0000 7fbae4d95640 1 -- 172.21.15.32:0/2803855148 wait complete. 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 Processor -- start 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 -- start start 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbae010db50 con 0x7fbae0103b70 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbae010d9d0 con 0x7fbae0101fc0 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbae4d95640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbae010dcd0 con 0x7fbae01031b0 2024-04-23T18:13:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48688/0 (socket says 172.21.15.32:48688) 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbade59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 -- 172.21.15.32:0/1814854640 learned_addr learned my addr 172.21.15.32:0/1814854640 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbad7fff640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 -- 172.21.15.32:0/1814854640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 msgr2=0x7fbae0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 -- 172.21.15.32:0/1814854640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 msgr2=0x7fbae0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbaded9d640 1 -- 172.21.15.32:0/1814854640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae019fd60 con 0x7fbae0103b70 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbade59c640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.034+0000 7fbad7fff640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbaded9d640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbad8033040 tx=0x7fbad80316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8003b70 con 0x7fbae0103b70 2024-04-23T18:13:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae019fff0 con 0x7fbae0103b70 2024-04-23T18:13:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad8003d10 con 0x7fbae0103b70 2024-04-23T18:13:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad802fb40 con 0x7fbae0103b70 2024-04-23T18:13:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.035+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae01a0500 con 0x7fbae0103b70 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.036+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae01092e0 con 0x7fbae0103b70 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.037+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbad8004610 con 0x7fbae0103b70 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.038+0000 7fbad77fe640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 0x7fbacc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.038+0000 7fbade59c640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 0x7fbacc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.038+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbad80f1530 con 0x7fbae0103b70 2024-04-23T18:13:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.038+0000 7fbade59c640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 0x7fbacc0a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbae0103010 tx=0x7fbad0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.041+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbad80be3b0 con 0x7fbae0103b70 2024-04-23T18:13:03.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.233+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbae002d7c0 con 0x7fbae0103b70 2024-04-23T18:13:03.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.233+0000 7fbad77fe640 1 -- 172.21.15.32:0/1814854640 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fbad80bdcb0 con 0x7fbae0103b70 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:13:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 msgr2=0x7fbacc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 0x7fbacc0a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbae0103010 tx=0x7fbad0005f90 comp rx=0 tx=0).stop 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 msgr2=0x7fbae019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbad8033040 tx=0x7fbad80316f0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 shutdown_connections 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae0103b70 0x7fbae019f6f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbae01031b0 0x7fbae0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbacc0a3bb0 0x7fbacc0a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.237+0000 7fbae4d95640 1 --2- 172.21.15.32:0/1814854640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbae0101fc0 0x7fbae0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.238+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 >> 172.21.15.32:0/1814854640 conn(0x7fbae00fd530 msgr2=0x7fbae01064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.238+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 shutdown_connections 2024-04-23T18:13:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.238+0000 7fbae4d95640 1 -- 172.21.15.32:0/1814854640 wait complete. 2024-04-23T18:13:03.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4140047476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2107980969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4212274339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:13:03.246 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T18:13:03.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 -- 172.21.15.32:0/3675214744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e8100860 msgr2=0x7f96e8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3675214744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e8100860 0x7f96e8100c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009900 tx=0x7f96e002a410 comp rx=0 tx=0).stop 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 -- 172.21.15.32:0/3675214744 shutdown_connections 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3675214744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8101b60 0x7f96e8101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3675214744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e81011c0 0x7f96e8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3675214744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e8100860 0x7f96e8100c80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 -- 172.21.15.32:0/3675214744 >> 172.21.15.32:0/3675214744 conn(0x7f96e80fc420 msgr2=0x7f96e80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 -- 172.21.15.32:0/3675214744 shutdown_connections 2024-04-23T18:13:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.360+0000 7f96ecb08640 1 -- 172.21.15.32:0/3675214744 wait complete. 2024-04-23T18:13:03.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 Processor -- start 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 -- start start 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8100860 0x7f96e819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e8101b60 0x7f96e81a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f96e810fce0 con 0x7f96e81011c0 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f96e810fb60 con 0x7f96e8101b60 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96ecb08640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96e810fe60 con 0x7f96e8100860 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.361+0000 7f96e5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48708/0 (socket says 172.21.15.32:48708) 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 -- 172.21.15.32:0/3192920154 learned_addr learned my addr 172.21.15.32:0/3192920154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e659c640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8100860 0x7f96e819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e6d9d640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e8101b60 0x7f96e81a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 -- 172.21.15.32:0/3192920154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8100860 msgr2=0x7f96e819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8100860 0x7f96e819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 -- 172.21.15.32:0/3192920154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e8101b60 msgr2=0x7f96e81a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e8101b60 0x7f96e81a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 -- 172.21.15.32:0/3192920154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81a2010 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96e5d9b640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ebc0 tx=0x7f96dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc010630 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96dc010c80 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e81a2300 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc018b60 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.362+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e81a2850 con 0x7f96e81011c0 2024-04-23T18:13:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.363+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e810b5b0 con 0x7f96e81011c0 2024-04-23T18:13:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.365+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f96dc018d00 con 0x7f96e81011c0 2024-04-23T18:13:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.365+0000 7f96d77fe640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 0x7f96d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.366+0000 7f96e659c640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 0x7f96d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.366+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f96dc0d05b0 con 0x7f96e81011c0 2024-04-23T18:13:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.366+0000 7f96e659c640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 0x7f96d00a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:03.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.370+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96dc09cd40 con 0x7f96e81011c0 2024-04-23T18:13:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4140047476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2107980969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4212274339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:13:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4140047476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2107980969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:03.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4212274339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:13:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.558+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f96e802d5c0 con 0x7f96e81011c0 2024-04-23T18:13:03.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.996+0000 7f96d77fe640 1 -- 172.21.15.32:0/3192920154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f96dc0a1040 con 0x7f96e81011c0 2024-04-23T18:13:03.991 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-04-23T18:13:03.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 msgr2=0x7f96d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 0x7f96d00a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e00314a0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 msgr2=0x7f96e819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:03.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ebc0 tx=0x7f96dc00c6a0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 shutdown_connections 2024-04-23T18:13:03.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f96d00a3bb0 0x7f96d00a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e8101b60 0x7f96e81a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96e81011c0 0x7f96e819a980 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 --2- 172.21.15.32:0/3192920154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e8100860 0x7f96e819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:03.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 >> 172.21.15.32:0/3192920154 conn(0x7f96e80fc420 msgr2=0x7f96e80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:03.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 shutdown_connections 2024-04-23T18:13:03.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:03.999+0000 7f96ecb08640 1 -- 172.21.15.32:0/3192920154 wait complete. 2024-04-23T18:13:04.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 -- 172.21.15.32:0/2223670661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 msgr2=0x7fe48810a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/2223670661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe48810a100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe480009ca0 tx=0x7fe48002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 -- 172.21.15.32:0/2223670661 shutdown_connections 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/2223670661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe48810a100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/2223670661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488105380 0x7fe488107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/2223670661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488102a50 0x7fe488104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 -- 172.21.15.32:0/2223670661 >> 172.21.15.32:0/2223670661 conn(0x7fe4880fc440 msgr2=0x7fe4880fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 -- 172.21.15.32:0/2223670661 shutdown_connections 2024-04-23T18:13:04.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.127+0000 7fe48d0ef640 1 -- 172.21.15.32:0/2223670661 wait complete. 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 Processor -- start 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 -- start start 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488102a50 0x7fe48819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488105380 0x7fe48819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe48810db10 con 0x7fe488107cb0 2024-04-23T18:13:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe48810d990 con 0x7fe488105380 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.128+0000 7fe48d0ef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe48810dc90 con 0x7fe488102a50 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48728/0 (socket says 172.21.15.32:48728) 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 -- 172.21.15.32:0/3660373606 learned_addr learned my addr 172.21.15.32:0/3660373606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe486d9d640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488102a50 0x7fe48819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48659c640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488105380 0x7fe48819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 -- 172.21.15.32:0/3660373606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488102a50 msgr2=0x7fe48819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488102a50 0x7fe48819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 -- 172.21.15.32:0/3660373606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488105380 msgr2=0x7fe48819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488105380 0x7fe48819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 -- 172.21.15.32:0/3660373606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4881a1fe0 con 0x7fe488107cb0 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe48759e640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4800097b0 tx=0x7fe480031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.129+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480003bf0 con 0x7fe488107cb0 2024-04-23T18:13:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.130+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe480003d90 con 0x7fe488107cb0 2024-04-23T18:13:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.130+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4881a2270 con 0x7fe488107cb0 2024-04-23T18:13:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.130+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48002fc70 con 0x7fe488107cb0 2024-04-23T18:13:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.130+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4881a27c0 con 0x7fe488107cb0 2024-04-23T18:13:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.131+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4881097a0 con 0x7fe488107cb0 2024-04-23T18:13:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.132+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe480004040 con 0x7fe488107cb0 2024-04-23T18:13:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.133+0000 7fe47ffff640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 0x7fe4740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.133+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fe4800f1820 con 0x7fe488107cb0 2024-04-23T18:13:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.133+0000 7fe486d9d640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 0x7fe4740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.133+0000 7fe486d9d640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 0x7fe4740a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe470009bd0 tx=0x7fe470005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:04.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.137+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4800be530 con 0x7fe488107cb0 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:04.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1814854640' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:13:04.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:04.246 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.247 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:04.247 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1814854640' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:13:04.247 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:13:04.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1814854640' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:13:04.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:13:04.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:04.326+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe48802d5c0 con 0x7fe488107cb0 2024-04-23T18:13:04.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.002+0000 7fe47ffff640 1 -- 172.21.15.32:0/3660373606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7fe4800bde30 con 0x7fe488107cb0 2024-04-23T18:13:04.996 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-04-23T18:13:05.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.006+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 msgr2=0x7fe4740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:05.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.006+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 0x7fe4740a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe470009bd0 tx=0x7fe470005f90 comp rx=0 tx=0).stop 2024-04-23T18:13:05.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.006+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 msgr2=0x7fe4881a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:05.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.006+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4800097b0 tx=0x7fe480031840 comp rx=0 tx=0).stop 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 shutdown_connections 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe488107cb0 0x7fe4881a1970 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe4740a3bb0 0x7fe4740a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe488105380 0x7fe48819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 --2- 172.21.15.32:0/3660373606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe488102a50 0x7fe48819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.008+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 >> 172.21.15.32:0/3660373606 conn(0x7fe4880fc440 msgr2=0x7fe4881021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:05.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.009+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 shutdown_connections 2024-04-23T18:13:05.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.009+0000 7fe48d0ef640 1 -- 172.21.15.32:0/3660373606 wait complete. 2024-04-23T18:13:05.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T18:13:05.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 -- 172.21.15.32:0/2669849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9154100e90 msgr2=0x7f91541012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 --2- 172.21.15.32:0/2669849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9154100e90 0x7f91541012b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9144009900 tx=0x7f914402a410 comp rx=0 tx=0).stop 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 -- 172.21.15.32:0/2669849921 shutdown_connections 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 --2- 172.21.15.32:0/2669849921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91541029a0 0x7f91541092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 --2- 172.21.15.32:0/2669849921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 --2- 172.21.15.32:0/2669849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9154100e90 0x7f91541012b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 -- 172.21.15.32:0/2669849921 >> 172.21.15.32:0/2669849921 conn(0x7f91540fc420 msgr2=0x7f91540fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.140+0000 7f915a956640 1 -- 172.21.15.32:0/2669849921 shutdown_connections 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 -- 172.21.15.32:0/2669849921 wait complete. 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 Processor -- start 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 -- start start 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9154100e90 0x7f9154198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f915410db30 con 0x7f91541029a0 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f915410d9b0 con 0x7f9154100e90 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.141+0000 7f915a956640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f915410dcb0 con 0x7f9154101fe0 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48746/0 (socket says 172.21.15.32:48746) 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f91537fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 -- 172.21.15.32:0/935248846 learned_addr learned my addr 172.21.15.32:0/935248846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9153fff640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9154100e90 0x7f9154198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 -- 172.21.15.32:0/935248846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 msgr2=0x7f9154198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 -- 172.21.15.32:0/935248846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9154100e90 msgr2=0x7f9154198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9154100e90 0x7f9154198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 -- 172.21.15.32:0/935248846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915419fe40 con 0x7f91541029a0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f91537fe640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f9158ef3640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f914c00cad0 tx=0x7f914c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c00ae20 con 0x7f91541029a0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914c005020 con 0x7f91541029a0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91541a0130 con 0x7f91541029a0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c00f6b0 con 0x7f91541029a0 2024-04-23T18:13:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.142+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541a0630 con 0x7f91541029a0 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.143+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9154108310 con 0x7f91541029a0 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.144+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f914c0051d0 con 0x7f91541029a0 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.145+0000 7f91517fa640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 0x7f913c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.145+0000 7f9153fff640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 0x7f913c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.145+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f914c0d0740 con 0x7f91541029a0 2024-04-23T18:13:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.145+0000 7f9153fff640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 0x7f913c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9144000c00 tx=0x7f91440314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:05.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.149+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f914c014030 con 0x7f91541029a0 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: pgmap v287: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:05.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:05.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: pgmap v287: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:05.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:05.374+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f915402d5c0 con 0x7f91541029a0 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3192920154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: pgmap v287: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:05.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:13:05.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.004+0000 7f91517fa640 1 -- 172.21.15.32:0/935248846 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f914c09d230 con 0x7f91541029a0 2024-04-23T18:13:05.998 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.006+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 msgr2=0x7f913c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.006+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 0x7f913c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9144000c00 tx=0x7f91440314a0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.006+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 msgr2=0x7f915419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.006+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f914c00cad0 tx=0x7f914c00cf90 comp rx=0 tx=0).stop 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 shutdown_connections 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91541029a0 0x7f915419f7d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9154101fe0 0x7f9154198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f913c0a3bb0 0x7f913c0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 --2- 172.21.15.32:0/935248846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9154100e90 0x7f9154198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 >> 172.21.15.32:0/935248846 conn(0x7f91540fc420 msgr2=0x7f91541054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 shutdown_connections 2024-04-23T18:13:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.007+0000 7f915a956640 1 -- 172.21.15.32:0/935248846 wait complete. 2024-04-23T18:13:06.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:13:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 -- 172.21.15.32:0/1253242079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f40103b60 msgr2=0x7f5f4010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 --2- 172.21.15.32:0/1253242079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f40103b60 0x7f5f4010a3f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009ca0 tx=0x7f5f3802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 -- 172.21.15.32:0/1253242079 shutdown_connections 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 --2- 172.21.15.32:0/1253242079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f40103b60 0x7f5f4010a3f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 --2- 172.21.15.32:0/1253242079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f401031a0 0x7f5f40103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 --2- 172.21.15.32:0/1253242079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f40101fb0 0x7f5f401023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 -- 172.21.15.32:0/1253242079 >> 172.21.15.32:0/1253242079 conn(0x7f5f400fd540 msgr2=0x7f5f400ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 -- 172.21.15.32:0/1253242079 shutdown_connections 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.132+0000 7f5f463e5640 1 -- 172.21.15.32:0/1253242079 wait complete. 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 Processor -- start 2024-04-23T18:13:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 -- start start 2024-04-23T18:13:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f40101fb0 0x7f5f401980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f40103b60 0x7f5f4019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010daf0 con 0x7f5f401031a0 2024-04-23T18:13:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010d970 con 0x7f5f40101fb0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f463e5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010dc70 con 0x7f5f40103b60 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48754/0 (socket says 172.21.15.32:48754) 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 -- 172.21.15.32:0/3391971345 learned_addr learned my addr 172.21.15.32:0/3391971345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 -- 172.21.15.32:0/3391971345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f40103b60 msgr2=0x7f5f4019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f40103b60 0x7f5f4019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 -- 172.21.15.32:0/3391971345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f40101fb0 msgr2=0x7f5f401980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f40101fb0 0x7f5f401980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.133+0000 7f5f3f7fe640 1 -- 172.21.15.32:0/3391971345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4019fcc0 con 0x7f5f401031a0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f3f7fe640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400c920 tx=0x7f5f3400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34007c70 con 0x7f5f401031a0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4019ffb0 con 0x7f5f401031a0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f34007e10 con 0x7f5f401031a0 2024-04-23T18:13:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34018820 con 0x7f5f401031a0 2024-04-23T18:13:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.134+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f401a0470 con 0x7f5f401031a0 2024-04-23T18:13:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.135+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f40109270 con 0x7f5f401031a0 2024-04-23T18:13:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.136+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5f3400f450 con 0x7f5f401031a0 2024-04-23T18:13:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.137+0000 7f5f3d7fa640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 0x7f5f280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.137+0000 7f5f3ffff640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 0x7f5f280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.137+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5f340d0510 con 0x7f5f401031a0 2024-04-23T18:13:06.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.137+0000 7f5f3ffff640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 0x7f5f280a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f30005fd0 tx=0x7f5f3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.141+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f34014030 con 0x7f5f401031a0 2024-04-23T18:13:06.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:13:06.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:13:06.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:13:06.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:13:06.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:13:06.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:06 smithi032 ceph-mon[29221]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:13:06.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.330+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f40062c50 con 0x7f5f401031a0 2024-04-23T18:13:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.331+0000 7f5f3d7fa640 1 -- 172.21.15.32:0/3391971345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22161 (secure 0 0 0) 0x7f5f3400de20 con 0x7f5f401031a0 2024-04-23T18:13:06.326 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:06.326 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:05.375894+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 msgr2=0x7f5f280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 0x7f5f280a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f30005fd0 tx=0x7f5f3000b410 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 msgr2=0x7f5f401985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400c920 tx=0x7f5f3400cde0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 shutdown_connections 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f40103b60 0x7f5f4019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f401031a0 0x7f5f401985e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5f280a3bb0 0x7f5f280a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.333+0000 7f5f463e5640 1 --2- 172.21.15.32:0/3391971345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f40101fb0 0x7f5f401980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.334+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 >> 172.21.15.32:0/3391971345 conn(0x7f5f400fd540 msgr2=0x7f5f40106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.334+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 shutdown_connections 2024-04-23T18:13:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.334+0000 7f5f463e5640 1 -- 172.21.15.32:0/3391971345 wait complete. 2024-04-23T18:13:06.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:13:06.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:06 smithi040 ceph-mon[32626]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3660373606' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:13:06.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:06 smithi064 ceph-mon[33062]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 -- 172.21.15.32:0/802212489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 msgr2=0x7f755010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 --2- 172.21.15.32:0/802212489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755010a0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7548009900 tx=0x7f754802a410 comp rx=0 tx=0).stop 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 -- 172.21.15.32:0/802212489 shutdown_connections 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 --2- 172.21.15.32:0/802212489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755010a0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 --2- 172.21.15.32:0/802212489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f75501077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 --2- 172.21.15.32:0/802212489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 0x7f7550104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 -- 172.21.15.32:0/802212489 >> 172.21.15.32:0/802212489 conn(0x7f75500fc420 msgr2=0x7f75500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 -- 172.21.15.32:0/802212489 shutdown_connections 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.446+0000 7f75554ef640 1 -- 172.21.15.32:0/802212489 wait complete. 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 Processor -- start 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 -- start start 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 0x7f75501981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f7550198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f755010db50 con 0x7f7550107cf0 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f755010d9d0 con 0x7f7550102a90 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f75554ef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f755010dcd0 con 0x7f75501053c0 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f754f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f754f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48766/0 (socket says 172.21.15.32:48766) 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f754f7fe640 1 -- 172.21.15.32:0/2253360936 learned_addr learned my addr 172.21.15.32:0/2253360936 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.447+0000 7f754effd640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 0x7f75501981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754e7fc640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f7550198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 -- 172.21.15.32:0/2253360936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 msgr2=0x7f7550198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f7550198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 -- 172.21.15.32:0/2253360936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 msgr2=0x7f75501981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 0x7f75501981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 -- 172.21.15.32:0/2253360936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755019fe00 con 0x7f7550107cf0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754e7fc640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f7550198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f754f7fe640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7548031500 tx=0x7f7548031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754803f030 con 0x7f7550107cf0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7548003aa0 con 0x7f7550107cf0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548035070 con 0x7f7550107cf0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75501a0090 con 0x7f7550107cf0 2024-04-23T18:13:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.448+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75501a0580 con 0x7f7550107cf0 2024-04-23T18:13:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.449+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75500fdf10 con 0x7f7550107cf0 2024-04-23T18:13:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.450+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7548042030 con 0x7f7550107cf0 2024-04-23T18:13:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.451+0000 7f7537fff640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 0x7f75380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.451+0000 7f754effd640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 0x7f75380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.451+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f75480f0f60 con 0x7f7550107cf0 2024-04-23T18:13:06.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.451+0000 7f754effd640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 0x7f75380a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f754000a400 tx=0x7f754000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.454+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75480bd9f0 con 0x7f7550107cf0 2024-04-23T18:13:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.642+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7550062b50 con 0x7f7550107cf0 2024-04-23T18:13:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.644+0000 7f7537fff640 1 -- 172.21.15.32:0/2253360936 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22148 (secure 0 0 0) 0x7f75480bd2f0 con 0x7f7550107cf0 2024-04-23T18:13:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:06.639 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:06.508628+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 msgr2=0x7f75380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 0x7f75380a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f754000a400 tx=0x7f754000d040 comp rx=0 tx=0).stop 2024-04-23T18:13:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 msgr2=0x7f755019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7548031500 tx=0x7f7548031a50 comp rx=0 tx=0).stop 2024-04-23T18:13:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 shutdown_connections 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7550107cf0 0x7f755019f790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75501053c0 0x7f7550198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f75380a3bb0 0x7f75380a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 --2- 172.21.15.32:0/2253360936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7550102a90 0x7f75501981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 >> 172.21.15.32:0/2253360936 conn(0x7f75500fc420 msgr2=0x7f75501033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 shutdown_connections 2024-04-23T18:13:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.646+0000 7f75554ef640 1 -- 172.21.15.32:0/2253360936 wait complete. 2024-04-23T18:13:06.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 -- 172.21.15.32:0/2990176866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068101ff0 msgr2=0x7f9068102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 --2- 172.21.15.32:0/2990176866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068101ff0 0x7f9068102470 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9058009900 tx=0x7f905802a410 comp rx=0 tx=0).stop 2024-04-23T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 -- 172.21.15.32:0/2990176866 shutdown_connections 2024-04-23T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 --2- 172.21.15.32:0/2990176866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 0x7f90681092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 --2- 172.21.15.32:0/2990176866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068101ff0 0x7f9068102470 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 --2- 172.21.15.32:0/2990176866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068100ea0 0x7f90681012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 -- 172.21.15.32:0/2990176866 >> 172.21.15.32:0/2990176866 conn(0x7f90680fc430 msgr2=0x7f90680fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 -- 172.21.15.32:0/2990176866 shutdown_connections 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.762+0000 7f906d7ad640 1 -- 172.21.15.32:0/2990176866 wait complete. 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 Processor -- start 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 -- start start 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068101ff0 0x7f9068198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 0x7f906819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f906810ca00 con 0x7f9068100ea0 2024-04-23T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f906810c880 con 0x7f90681029b0 2024-04-23T18:13:06.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f906d7ad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f906810cb80 con 0x7f9068101ff0 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43262/0 (socket says 172.21.15.32:43262) 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 -- 172.21.15.32:0/3646451875 learned_addr learned my addr 172.21.15.32:0/3646451875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f90667fc640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068101ff0 0x7f9068198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f90677fe640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 0x7f906819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 -- 172.21.15.32:0/3646451875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068101ff0 msgr2=0x7f9068198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068101ff0 0x7f9068198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 -- 172.21.15.32:0/3646451875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 msgr2=0x7f906819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 0x7f906819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.763+0000 7f9066ffd640 1 -- 172.21.15.32:0/3646451875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906819fe50 con 0x7f9068100ea0 2024-04-23T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f9066ffd640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9f0 tx=0x7f905c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c00cd80 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f905c0054d0 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90681a0140 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c010690 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.764+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90681a0660 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.765+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90681a0920 con 0x7f9068100ea0 2024-04-23T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.766+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f905c004e90 con 0x7f9068100ea0 2024-04-23T18:13:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.766+0000 7f904ffff640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 0x7f90500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.767+0000 7f90667fc640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 0x7f90500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.767+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f905c0d0790 con 0x7f9068100ea0 2024-04-23T18:13:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.767+0000 7f90667fc640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 0x7f90500a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9058009230 tx=0x7f9058031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.771+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f905c09d220 con 0x7f9068100ea0 2024-04-23T18:13:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:06.995+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f90681a0af0 con 0x7f9068100ea0 2024-04-23T18:13:07.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.014+0000 7f904ffff640 1 -- 172.21.15.32:0/3646451875 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f905c09cb20 con 0x7f9068100ea0 2024-04-23T18:13:07.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 msgr2=0x7f90500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 0x7f90500a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9058009230 tx=0x7f9058031400 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 msgr2=0x7f9068198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9f0 tx=0x7f905c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 shutdown_connections 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90500a3c30 0x7f90500a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90681029b0 0x7f906819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9068101ff0 0x7f9068198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 --2- 172.21.15.32:0/3646451875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9068100ea0 0x7f9068198230 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 >> 172.21.15.32:0/3646451875 conn(0x7f90680fc430 msgr2=0x7f90680fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 shutdown_connections 2024-04-23T18:13:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.016+0000 7f906d7ad640 1 -- 172.21.15.32:0/3646451875 wait complete. 2024-04-23T18:13:07.021 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:13:07.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 -- 172.21.15.32:0/1708024344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 msgr2=0x7f21d41092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 --2- 172.21.15.32:0/1708024344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d41092d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009ca0 tx=0x7f21cc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 -- 172.21.15.32:0/1708024344 shutdown_connections 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 --2- 172.21.15.32:0/1708024344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d41092d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 --2- 172.21.15.32:0/1708024344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 --2- 172.21.15.32:0/1708024344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 0x7f21d41012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 -- 172.21.15.32:0/1708024344 >> 172.21.15.32:0/1708024344 conn(0x7f21d40fc430 msgr2=0x7f21d40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 -- 172.21.15.32:0/1708024344 shutdown_connections 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.132+0000 7f21da3c6640 1 -- 172.21.15.32:0/1708024344 wait complete. 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 Processor -- start 2024-04-23T18:13:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 -- start start 2024-04-23T18:13:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 0x7f21d410b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d410a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410db40 con 0x7f21d41029b0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d9c0 con 0x7f21d4100ea0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21da3c6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410dcc0 con 0x7f21d4101ff0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d8963640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d410a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 0x7f21d410b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:50422/0 (socket says 172.21.15.32:50422) 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d8963640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d410a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43290/0 (socket says 172.21.15.32:43290) 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.133+0000 7f21d37fe640 1 -- 172.21.15.32:0/2564318888 learned_addr learned my addr 172.21.15.32:0/2564318888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 -- 172.21.15.32:0/2564318888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 msgr2=0x7f21d410b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 0x7f21d410b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 -- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 msgr2=0x7f21d410a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d410a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 -- 172.21.15.32:0/2564318888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d410ac80 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d37fe640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21c800d950 tx=0x7f21c800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8017020 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d41a22f0 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21c8009c30 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c800f600 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.134+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d41a2860 con 0x7f21d4101ff0 2024-04-23T18:13:07.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.135+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d41081b0 con 0x7f21d4101ff0 2024-04-23T18:13:07.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.136+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f21c801a020 con 0x7f21d4101ff0 2024-04-23T18:13:07.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.137+0000 7f21d17fa640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 0x7f21bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:07.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.137+0000 7f21d3fff640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 0x7f21bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:07.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.137+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f21c80d0440 con 0x7f21d4101ff0 2024-04-23T18:13:07.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.137+0000 7f21d3fff640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 0x7f21bc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005d00 tx=0x7f21c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.141+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21c8021020 con 0x7f21d4101ff0 2024-04-23T18:13:07.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: pgmap v290: 97 pgs: 10 creating+peering, 76 unknown, 11 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3391971345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2253360936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:07 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:13:07.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:07.369+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f21d402d810 con 0x7f21d4101ff0 2024-04-23T18:13:07.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:13:07.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:13:07.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:13:07.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: pgmap v290: 97 pgs: 10 creating+peering, 76 unknown, 11 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:07.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3391971345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:13:07.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2253360936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:07 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/935248846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: pgmap v290: 97 pgs: 10 creating+peering, 76 unknown, 11 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3391971345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:13:07.492 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2253360936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:07.492 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:07 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:13:08.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.026+0000 7f21d17fa640 1 -- 172.21.15.32:0/2564318888 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f21c809cf70 con 0x7f21d4101ff0 2024-04-23T18:13:08.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 msgr2=0x7f21bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:08.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 0x7f21bc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005d00 tx=0x7f21c400b410 comp rx=0 tx=0).stop 2024-04-23T18:13:08.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 msgr2=0x7f21d4109e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:08.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21c800d950 tx=0x7f21c800de10 comp rx=0 tx=0).stop 2024-04-23T18:13:08.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 shutdown_connections 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d41029b0 0x7f21d410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21d4101ff0 0x7f21d4109e80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f21bc0a3bb0 0x7f21bc0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 --2- 172.21.15.32:0/2564318888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d4100ea0 0x7f21d410b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 >> 172.21.15.32:0/2564318888 conn(0x7f21d40fc430 msgr2=0x7f21d4105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 shutdown_connections 2024-04-23T18:13:08.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.029+0000 7f21da3c6640 1 -- 172.21.15.32:0/2564318888 wait complete. 2024-04-23T18:13:08.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1491775211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 msgr2=0x7f3e08102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1491775211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e08102470 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009900 tx=0x7f3dfc02a410 comp rx=0 tx=0).stop 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1491775211 shutdown_connections 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1491775211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e081029b0 0x7f3e081092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1491775211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e08102470 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1491775211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e08100ea0 0x7f3e081012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1491775211 >> 172.21.15.32:0/1491775211 conn(0x7f3e080fc430 msgr2=0x7f3e080fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1491775211 shutdown_connections 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1491775211 wait complete. 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.160+0000 7f3e0d53d640 1 Processor -- start 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 -- start start 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e08100ea0 0x7f3e08198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 0x7f3e0819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0810ca40 con 0x7f3e08101ff0 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0810c8c0 con 0x7f3e081029b0 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0810cbc0 con 0x7f3e08100ea0 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43314/0 (socket says 172.21.15.32:43314) 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 -- 172.21.15.32:0/1734785427 learned_addr learned my addr 172.21.15.32:0/1734785427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e06ffd640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e08100ea0 0x7f3e08198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e077fe640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 0x7f3e0819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 -- 172.21.15.32:0/1734785427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e08100ea0 msgr2=0x7f3e08198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e08100ea0 0x7f3e08198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 -- 172.21.15.32:0/1734785427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 msgr2=0x7f3e0819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 0x7f3e0819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 -- 172.21.15.32:0/1734785427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0819fe90 con 0x7f3e08101ff0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e077fe640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 0x7f3e0819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e067fc640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc031440 tx=0x7f3dfc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc03f070 con 0x7f3e08101ff0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.161+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e081a0120 con 0x7f3e08101ff0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.162+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dfc031c50 con 0x7f3e08101ff0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.162+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc035030 con 0x7f3e08101ff0 2024-04-23T18:13:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.162+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e081a0660 con 0x7f3e08101ff0 2024-04-23T18:13:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.162+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e081a08c0 con 0x7f3e08101ff0 2024-04-23T18:13:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.164+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3dfc042070 con 0x7f3e08101ff0 2024-04-23T18:13:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.164+0000 7f3deffff640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 0x7f3df00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.164+0000 7f3e06ffd640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 0x7f3df00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.164+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3dfc0f0ab0 con 0x7f3e08101ff0 2024-04-23T18:13:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.165+0000 7f3e06ffd640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 0x7f3df00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3df8005d00 tx=0x7f3df800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.169+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3dfc0bd5e0 con 0x7f3e08101ff0 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: Cluster is now healthy 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:active 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:08.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2564318888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:08 smithi040 ceph-mon[32626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:08.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:08.397+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f3e081a0a90 con 0x7f3e08101ff0 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: Cluster is now healthy 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:active 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2564318888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:08 smithi064 ceph-mon[33062]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: Cluster is now healthy 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3646451875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] up:active 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2564318888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:08.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:08 smithi032 ceph-mon[29221]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:13:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.031+0000 7f3deffff640 1 -- 172.21.15.32:0/1734785427 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f3dfc0bcee0 con 0x7f3e08101ff0 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 msgr2=0x7f3df00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 0x7f3df00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3df8005d00 tx=0x7f3df800b410 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 msgr2=0x7f3e081987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc031440 tx=0x7f3dfc003c10 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 shutdown_connections 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3df00a3c30 0x7f3df00a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e081029b0 0x7f3e0819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e08101ff0 0x7f3e081987b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 --2- 172.21.15.32:0/1734785427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e08100ea0 0x7f3e08198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 >> 172.21.15.32:0/1734785427 conn(0x7f3e080fc430 msgr2=0x7f3e080fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 shutdown_connections 2024-04-23T18:13:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.034+0000 7f3e0d53d640 1 -- 172.21.15.32:0/1734785427 wait complete. 2024-04-23T18:13:09.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T18:13:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- 172.21.15.32:0/1459853317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 msgr2=0x7fde40102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 --2- 172.21.15.32:0/1459853317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde40102470 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fde34009900 tx=0x7fde3402a410 comp rx=0 tx=0).stop 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- 172.21.15.32:0/1459853317 shutdown_connections 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 --2- 172.21.15.32:0/1459853317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde401092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 --2- 172.21.15.32:0/1459853317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde40102470 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 --2- 172.21.15.32:0/1459853317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 0x7fde401012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- 172.21.15.32:0/1459853317 >> 172.21.15.32:0/1459853317 conn(0x7fde400fc430 msgr2=0x7fde400fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- 172.21.15.32:0/1459853317 shutdown_connections 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- 172.21.15.32:0/1459853317 wait complete. 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 Processor -- start 2024-04-23T18:13:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.162+0000 7fde465ad640 1 -- start start 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 0x7fde4006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde4019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010ca00 con 0x7fde40101ff0 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010c880 con 0x7fde401029b0 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde465ad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010cb80 con 0x7fde40100ea0 2024-04-23T18:13:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43322/0 (socket says 172.21.15.32:43322) 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde44b4a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde4019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 -- 172.21.15.32:0/571961686 learned_addr learned my addr 172.21.15.32:0/571961686 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3ffff640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 0x7fde4006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 -- 172.21.15.32:0/571961686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 msgr2=0x7fde4006b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 0x7fde4006b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 -- 172.21.15.32:0/571961686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 msgr2=0x7fde4019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde4019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde3f7fe640 1 -- 172.21.15.32:0/571961686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4019fe60 con 0x7fde40101ff0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.163+0000 7fde44b4a640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde4019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde3f7fe640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fde34006010 tx=0x7fde340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3403f030 con 0x7fde40101ff0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde401a00f0 con 0x7fde40101ff0 2024-04-23T18:13:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340318f0 con 0x7fde40101ff0 2024-04-23T18:13:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34035070 con 0x7fde40101ff0 2024-04-23T18:13:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.164+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde401a0630 con 0x7fde40101ff0 2024-04-23T18:13:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.165+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde40108230 con 0x7fde40101ff0 2024-04-23T18:13:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.166+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fde34042030 con 0x7fde40101ff0 2024-04-23T18:13:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.167+0000 7fde3d7fa640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 0x7fde280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:09.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.167+0000 7fde3ffff640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 0x7fde280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:09.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.167+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fde340f0b50 con 0x7fde40101ff0 2024-04-23T18:13:09.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.167+0000 7fde3ffff640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 0x7fde280a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde30005fd0 tx=0x7fde3000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.171+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde340bd030 con 0x7fde40101ff0 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: pgmap v292: 97 pgs: 10 creating+peering, 33 unknown, 54 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-04-23T18:13:09.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:09 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:13:09.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:09.395+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fde4002d5c0 con 0x7fde40101ff0 2024-04-23T18:13:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:13:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:13:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:13:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:13:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:13:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: pgmap v292: 97 pgs: 10 creating+peering, 33 unknown, 54 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-04-23T18:13:09.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:09 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:13:09.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: pgmap v292: 97 pgs: 10 creating+peering, 33 unknown, 54 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-04-23T18:13:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:09 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:13:10.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.038+0000 7fde3d7fa640 1 -- 172.21.15.32:0/571961686 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fde340c1330 con 0x7fde40101ff0 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.041+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 msgr2=0x7fde280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.041+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 0x7fde280a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde30005fd0 tx=0x7fde3000b390 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.041+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 msgr2=0x7fde4019cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.041+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fde34006010 tx=0x7fde340042f0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 shutdown_connections 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde280a3bb0 0x7fde280a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde401029b0 0x7fde4019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde40101ff0 0x7fde4019cf30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 --2- 172.21.15.32:0/571961686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde40100ea0 0x7fde4006b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 >> 172.21.15.32:0/571961686 conn(0x7fde400fc430 msgr2=0x7fde40105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 shutdown_connections 2024-04-23T18:13:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.042+0000 7fde465ad640 1 -- 172.21.15.32:0/571961686 wait complete. 2024-04-23T18:13:10.050 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-23T18:13:10.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-23T18:13:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 -- 172.21.15.32:0/2560947349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8107d40 msgr2=0x7fc9a810a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 --2- 172.21.15.32:0/2560947349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8107d40 0x7fc9a810a130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009ca0 tx=0x7fc9a002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 -- 172.21.15.32:0/2560947349 shutdown_connections 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 --2- 172.21.15.32:0/2560947349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8107d40 0x7fc9a810a130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 --2- 172.21.15.32:0/2560947349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8105410 0x7fc9a8107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 --2- 172.21.15.32:0/2560947349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a8104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 -- 172.21.15.32:0/2560947349 >> 172.21.15.32:0/2560947349 conn(0x7fc9a80fc430 msgr2=0x7fc9a80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 -- 172.21.15.32:0/2560947349 shutdown_connections 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.169+0000 7fc9af657640 1 -- 172.21.15.32:0/2560947349 wait complete. 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 Processor -- start 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 -- start start 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8107d40 0x7fc9a819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a810db40 con 0x7fc9a8105410 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a810d9c0 con 0x7fc9a8107d40 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9af657640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a810dcc0 con 0x7fc9a8102ae0 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43342/0 (socket says 172.21.15.32:43342) 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 -- 172.21.15.32:0/1023112640 learned_addr learned my addr 172.21.15.32:0/1023112640 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9adbf4640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8107d40 0x7fc9a819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9ad3f3640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 -- 172.21.15.32:0/1023112640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 msgr2=0x7fc9a81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 -- 172.21.15.32:0/1023112640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8107d40 msgr2=0x7fc9a819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8107d40 0x7fc9a819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 -- 172.21.15.32:0/1023112640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a819fdd0 con 0x7fc9a8105410 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9ad3f3640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.170+0000 7fc9acbf2640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e9a0 tx=0x7fc99800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.171+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99800cd80 con 0x7fc9a8105410 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.171+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9980035f0 con 0x7fc9a8105410 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.171+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998010690 con 0x7fc9a8105410 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.171+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a81a00c0 con 0x7fc9a8105410 2024-04-23T18:13:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.171+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a81a0660 con 0x7fc9a8105410 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.172+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a80fdf20 con 0x7fc9a8105410 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.173+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc998004000 con 0x7fc9a8105410 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.174+0000 7fc99e7fc640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 0x7fc9940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.174+0000 7fc9ad3f3640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 0x7fc9940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.174+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc9980d0530 con 0x7fc9a8105410 2024-04-23T18:13:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.174+0000 7fc9ad3f3640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 0x7fc9940a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40058c0 tx=0x7fc9a4005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.178+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9980042f0 con 0x7fc9a8105410 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:active 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: Cluster is now healthy 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:13:10.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:13:10.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.395+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc9a8062390 con 0x7fc9940a3c30 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:active 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: Cluster is now healthy 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:13:10.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:13:10.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:13:10.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.422+0000 7fc99e7fc640 1 -- 172.21.15.32:0/1023112640 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc9a8062390 con 0x7fc9940a3c30 2024-04-23T18:13:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 msgr2=0x7fc9940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 0x7fc9940a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40058c0 tx=0x7fc9a4005d30 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 msgr2=0x7fc9a81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e9a0 tx=0x7fc99800ee60 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 shutdown_connections 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc9940a3c30 0x7fc9940a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a8107d40 0x7fc9a819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9a8105410 0x7fc9a81986f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 --2- 172.21.15.32:0/1023112640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9a8102ae0 0x7fc9a81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 >> 172.21.15.32:0/1023112640 conn(0x7fc9a80fc430 msgr2=0x7fc9a8103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 shutdown_connections 2024-04-23T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.424+0000 7fc9af657640 1 -- 172.21.15.32:0/1023112640 wait complete. 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1734785427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] up:active 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: Cluster is now healthy 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:13:10.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:10 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:13:10.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-23T18:13:10.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 -- 172.21.15.32:0/1971215059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d8107d00 msgr2=0x7f35d810a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/1971215059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d8107d00 0x7f35d810a0f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009ca0 tx=0x7f35d002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 -- 172.21.15.32:0/1971215059 shutdown_connections 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/1971215059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d8107d00 0x7f35d810a0f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/1971215059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35d81053d0 0x7f35d81077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/1971215059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d8104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 -- 172.21.15.32:0/1971215059 >> 172.21.15.32:0/1971215059 conn(0x7f35d80fc410 msgr2=0x7f35d80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 -- 172.21.15.32:0/1971215059 shutdown_connections 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.542+0000 7f35dcb7d640 1 -- 172.21.15.32:0/1971215059 wait complete. 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 Processor -- start 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 -- start start 2024-04-23T18:13:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35d8107d00 0x7f35d81a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810dae0 con 0x7f35d81053d0 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810d960 con 0x7f35d8102aa0 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35dcb7d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810dc60 con 0x7f35d8107d00 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43370/0 (socket says 172.21.15.32:43370) 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 -- 172.21.15.32:0/576053897 learned_addr learned my addr 172.21.15.32:0/576053897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 -- 172.21.15.32:0/576053897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35d8107d00 msgr2=0x7f35d81a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35d8107d00 0x7f35d81a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 -- 172.21.15.32:0/576053897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 msgr2=0x7f35d819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d5d9b640 1 -- 172.21.15.32:0/576053897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d81a2000 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.543+0000 7f35d659c640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35d5d9b640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005fd0 tx=0x7f35cc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc017070 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35cc009c30 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d81a22f0 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc003d30 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.544+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d81a2840 con 0x7f35d81053d0 2024-04-23T18:13:10.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.545+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d81097f0 con 0x7f35d81053d0 2024-04-23T18:13:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.547+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f35cc003ed0 con 0x7f35d81053d0 2024-04-23T18:13:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.547+0000 7f35c77fe640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 0x7f35c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.547+0000 7f35d659c640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 0x7f35c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.547+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f35cc0d06c0 con 0x7f35d81053d0 2024-04-23T18:13:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.548+0000 7f35d659c640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 0x7f35c00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009fc0 tx=0x7f35c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.551+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35cc020030 con 0x7f35d81053d0 2024-04-23T18:13:10.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:13:10.400+0000 7f79bf4e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:13:10.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:13:10.400+0000 7f79bf4e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:13:10.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:13:10.400+0000 7f79bf4e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:13:10.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:13:10.400+0000 7f79bf4e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:13:10.758 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:13:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:13:10.400+0000 7f79bf4e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.766+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f35d8062420 con 0x7f35c00a3bb0 2024-04-23T18:13:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.790+0000 7f35c77fe640 1 -- 172.21.15.32:0/576053897 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f35d8062420 con 0x7f35c00a3bb0 2024-04-23T18:13:10.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 msgr2=0x7f35c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 0x7f35c00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009fc0 tx=0x7f35c8005f90 comp rx=0 tx=0).stop 2024-04-23T18:13:10.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 msgr2=0x7f35d819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005fd0 tx=0x7f35cc00dde0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 shutdown_connections 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35d8107d00 0x7f35d81a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35d81053d0 0x7f35d819a920 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f35c00a3bb0 0x7f35c00a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 --2- 172.21.15.32:0/576053897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d8102aa0 0x7f35d819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 >> 172.21.15.32:0/576053897 conn(0x7f35d80fc410 msgr2=0x7f35d8102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 shutdown_connections 2024-04-23T18:13:10.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.792+0000 7f35dcb7d640 1 -- 172.21.15.32:0/576053897 wait complete. 2024-04-23T18:13:10.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-23T18:13:10.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 -- 172.21.15.32:0/3420497478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8103b50 msgr2=0x7fdef810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 --2- 172.21.15.32:0/3420497478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8103b50 0x7fdef810a3e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009ca0 tx=0x7fdef002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 -- 172.21.15.32:0/3420497478 shutdown_connections 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 --2- 172.21.15.32:0/3420497478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8103b50 0x7fdef810a3e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 --2- 172.21.15.32:0/3420497478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103190 0x7fdef8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 --2- 172.21.15.32:0/3420497478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8101fa0 0x7fdef81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 -- 172.21.15.32:0/3420497478 >> 172.21.15.32:0/3420497478 conn(0x7fdef80fd510 msgr2=0x7fdef80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 -- 172.21.15.32:0/3420497478 shutdown_connections 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.911+0000 7fdefe459640 1 -- 172.21.15.32:0/3420497478 wait complete. 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 Processor -- start 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 -- start start 2024-04-23T18:13:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 0x7fdef81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103b50 0x7fdef819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810dae0 con 0x7fdef8101fa0 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810d960 con 0x7fdef8103190 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdefe459640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810dc60 con 0x7fdef8103b50 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43390/0 (socket says 172.21.15.32:43390) 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdef7fff640 1 -- 172.21.15.32:0/3050023647 learned_addr learned my addr 172.21.15.32:0/3050023647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.912+0000 7fdef77fe640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 0x7fdef81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdefc9f6640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103b50 0x7fdef819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 -- 172.21.15.32:0/3050023647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103b50 msgr2=0x7fdef819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103b50 0x7fdef819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 -- 172.21.15.32:0/3050023647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 msgr2=0x7fdef81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 0x7fdef81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 -- 172.21.15.32:0/3050023647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef819fdb0 con 0x7fdef8101fa0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef77fe640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 0x7fdef81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef7fff640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e9c0 tx=0x7fdee800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee800cde0 con 0x7fdef8101fa0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef81a00a0 con 0x7fdef8101fa0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee80108e0 con 0x7fdef8101fa0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80188d0 con 0x7fdef8101fa0 2024-04-23T18:13:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.913+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef81a0640 con 0x7fdef8101fa0 2024-04-23T18:13:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.914+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef8109340 con 0x7fdef8101fa0 2024-04-23T18:13:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.915+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdee801f020 con 0x7fdef8101fa0 2024-04-23T18:13:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.916+0000 7fdef57fa640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 0x7fdee00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.916+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdee80d0ad0 con 0x7fdef8101fa0 2024-04-23T18:13:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.916+0000 7fdef77fe640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 0x7fdee00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.917+0000 7fdef77fe640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 0x7fdee00a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdef81996a0 tx=0x7fdeec00a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:10.920+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdee809d600 con 0x7fdef8101fa0 2024-04-23T18:13:11.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.143+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fdef8062420 con 0x7fdee00a3bb0 2024-04-23T18:13:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.148+0000 7fdef57fa640 1 -- 172.21.15.32:0/3050023647 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fdef8062420 con 0x7fdee00a3bb0 2024-04-23T18:13:11.142 INFO:teuthology.orchestra.run.smithi032.stdout:/volumes/qa/sv_0/ad7ab546-19e6-488d-8390-9b577bb0df94 2024-04-23T18:13:11.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 msgr2=0x7fdee00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 0x7fdee00a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdef81996a0 tx=0x7fdeec00a250 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 msgr2=0x7fdef8198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e9c0 tx=0x7fdee800ee80 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 shutdown_connections 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdef8103b50 0x7fdef819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdee00a3bb0 0x7fdee00a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdef8103190 0x7fdef81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 --2- 172.21.15.32:0/3050023647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef8101fa0 0x7fdef8198190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 >> 172.21.15.32:0/3050023647 conn(0x7fdef80fd510 msgr2=0x7fdef8106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 shutdown_connections 2024-04-23T18:13:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.150+0000 7fdefe459640 1 -- 172.21.15.32:0/3050023647 wait complete. 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:active 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: pgmap v293: 97 pgs: 7 creating+peering, 90 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 10 op/s 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:11 smithi032 ceph-mon[29221]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- 172.21.15.32:0/2337971667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 msgr2=0x7fcf7410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/2337971667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf7410a0c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009ca0 tx=0x7fcf6c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- 172.21.15.32:0/2337971667 shutdown_connections 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/2337971667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf7410a0c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/2337971667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 0x7fcf74107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/2337971667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 0x7fcf74104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- 172.21.15.32:0/2337971667 >> 172.21.15.32:0/2337971667 conn(0x7fcf740fc420 msgr2=0x7fcf740fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- 172.21.15.32:0/2337971667 shutdown_connections 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- 172.21.15.32:0/2337971667 wait complete. 2024-04-23T18:13:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 Processor -- start 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 -- start start 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.274+0000 7fcf7aecb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 0x7fcf74102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 0x7fcf741009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7410dad0 con 0x7fcf74107c70 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7410d950 con 0x7fcf74102a10 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7410dc50 con 0x7fcf74105340 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43410/0 (socket says 172.21.15.32:43410) 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 -- 172.21.15.32:0/1239554038 learned_addr learned my addr 172.21.15.32:0/1239554038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf78c67640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 0x7fcf74102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf73fff640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 0x7fcf741009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 -- 172.21.15.32:0/1239554038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 msgr2=0x7fcf741009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 0x7fcf741009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 -- 172.21.15.32:0/1239554038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 msgr2=0x7fcf74102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 0x7fcf74102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 -- 172.21.15.32:0/1239554038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf74101790 con 0x7fcf74107c70 2024-04-23T18:13:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf79468640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c0097b0 tx=0x7fcf6c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c004240 con 0x7fcf74107c70 2024-04-23T18:13:11.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf6c0317c0 con 0x7fcf74107c70 2024-04-23T18:13:11.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf741a22f0 con 0x7fcf74107c70 2024-04-23T18:13:11.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.275+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c02fd80 con 0x7fcf74107c70 2024-04-23T18:13:11.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.276+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf741a2840 con 0x7fcf74107c70 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.277+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf740fdf10 con 0x7fcf74107c70 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.278+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf6c0043e0 con 0x7fcf74107c70 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.278+0000 7fcf71ffb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 0x7fcf640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.278+0000 7fcf78c67640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 0x7fcf640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.278+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fcf6c0f1b40 con 0x7fcf74107c70 2024-04-23T18:13:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.279+0000 7fcf78c67640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 0x7fcf640a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcf68005d00 tx=0x7fcf6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:11.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.283+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf6c0be670 con 0x7fcf74107c70 2024-04-23T18:13:11.367 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: 2024-04-23T18:13:10.927+0000 7fb17058d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T18:13:11.367 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: 2024-04-23T18:13:10.927+0000 7fb17058d700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-23T18:13:11.367 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:13:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: 2024-04-23T18:13:10.927+0000 7fb17058d700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-23T18:13:11.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:active 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: pgmap v293: 97 pgs: 7 creating+peering, 90 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 10 op/s 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:11 smithi040 ceph-mon[32626]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/571961686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] up:active 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: pgmap v293: 97 pgs: 7 creating+peering, 90 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 10 op/s 2024-04-23T18:13:11.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:11 smithi064 ceph-mon[33062]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:11.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.518+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcf74062390 con 0x7fcf640a3bb0 2024-04-23T18:13:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.543+0000 7fcf71ffb640 1 -- 172.21.15.32:0/1239554038 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcf74062390 con 0x7fcf640a3bb0 2024-04-23T18:13:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 msgr2=0x7fcf640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 0x7fcf640a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcf68005d00 tx=0x7fcf6800b410 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 msgr2=0x7fcf74100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c0097b0 tx=0x7fcf6c003c20 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 shutdown_connections 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf74107c70 0x7fcf74100f00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf74105340 0x7fcf741009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fcf640a3bb0 0x7fcf640a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 --2- 172.21.15.32:0/1239554038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf74102a10 0x7fcf74102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 >> 172.21.15.32:0/1239554038 conn(0x7fcf740fc420 msgr2=0x7fcf74103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 shutdown_connections 2024-04-23T18:13:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.545+0000 7fcf7aecb640 1 -- 172.21.15.32:0/1239554038 wait complete. 2024-04-23T18:13:11.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 -- 172.21.15.32:0/1668190158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 msgr2=0x7f47601023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 --2- 172.21.15.32:0/1668190158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f47601023c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4748009900 tx=0x7f474802a410 comp rx=0 tx=0).stop 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 -- 172.21.15.32:0/1668190158 shutdown_connections 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 --2- 172.21.15.32:0/1668190158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 --2- 172.21.15.32:0/1668190158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 0x7f4760103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 --2- 172.21.15.32:0/1668190158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f47601023c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 -- 172.21.15.32:0/1668190158 >> 172.21.15.32:0/1668190158 conn(0x7f47600fd510 msgr2=0x7f47600ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 -- 172.21.15.32:0/1668190158 shutdown_connections 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.666+0000 7f4766259640 1 -- 172.21.15.32:0/1668190158 wait complete. 2024-04-23T18:13:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 Processor -- start 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 -- start start 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 0x7f4760198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f476010db20 con 0x7f4760101fa0 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f476010d9a0 con 0x7f4760103190 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f4766259640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f476010dca0 con 0x7f4760103b50 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f475f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f475f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43426/0 (socket says 172.21.15.32:43426) 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.667+0000 7f475f7fe640 1 -- 172.21.15.32:0/2633801565 learned_addr learned my addr 172.21.15.32:0/2633801565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475ffff640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475effd640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 0x7f4760198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 -- 172.21.15.32:0/2633801565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 msgr2=0x7f476019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 -- 172.21.15.32:0/2633801565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 msgr2=0x7f4760198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 0x7f4760198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 -- 172.21.15.32:0/2633801565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476019fe40 con 0x7f4760101fa0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475ffff640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475f7fe640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4748006010 tx=0x7f4748004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474803f030 con 0x7f4760101fa0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4748003cb0 con 0x7f4760101fa0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47601a00d0 con 0x7f4760101fa0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748035070 con 0x7f4760101fa0 2024-04-23T18:13:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.668+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47601a0690 con 0x7f4760101fa0 2024-04-23T18:13:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.669+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47601092e0 con 0x7f4760101fa0 2024-04-23T18:13:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.670+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4748042030 con 0x7f4760101fa0 2024-04-23T18:13:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.671+0000 7f475cff9640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 0x7f474c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.671+0000 7f475effd640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 0x7f474c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:11.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.671+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f47480f09f0 con 0x7f4760101fa0 2024-04-23T18:13:11.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.671+0000 7f475effd640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 0x7f474c0a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4750005e10 tx=0x7f4750005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.675+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4748003e60 con 0x7f4760101fa0 2024-04-23T18:13:11.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.891+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4760062420 con 0x7f474c0a3c30 2024-04-23T18:13:11.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.898+0000 7f475cff9640 1 -- 172.21.15.32:0/2633801565 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4760062420 con 0x7f474c0a3c30 2024-04-23T18:13:11.893 INFO:teuthology.orchestra.run.smithi032.stdout:/volumes/qa/sv_1/d17f95c9-36ff-4b3f-b9ea-0f18c979081d 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 msgr2=0x7f474c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 0x7f474c0a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4750005e10 tx=0x7f4750005da0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 msgr2=0x7f4760198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4748006010 tx=0x7f4748004430 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 shutdown_connections 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4760103b50 0x7f476019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f474c0a3c30 0x7f474c0a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4760103190 0x7f4760198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 --2- 172.21.15.32:0/2633801565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4760101fa0 0x7f4760198220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.900+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 >> 172.21.15.32:0/2633801565 conn(0x7f47600fd510 msgr2=0x7f47601064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.901+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 shutdown_connections 2024-04-23T18:13:11.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:11.901+0000 7f4766259640 1 -- 172.21.15.32:0/2633801565 wait complete. 2024-04-23T18:13:11.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:13:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- 172.21.15.32:0/442190159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102a00 msgr2=0x7fca9c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/442190159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102a00 0x7fca9c109320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fca94009ca0 tx=0x7fca9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- 172.21.15.32:0/442190159 shutdown_connections 2024-04-23T18:13:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/442190159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102a00 0x7fca9c109320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/442190159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c102040 0x7fca9c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/442190159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c100e60 0x7fca9c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- 172.21.15.32:0/442190159 >> 172.21.15.32:0/442190159 conn(0x7fca9c0fc420 msgr2=0x7fca9c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- 172.21.15.32:0/442190159 shutdown_connections 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- 172.21.15.32:0/442190159 wait complete. 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 Processor -- start 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.015+0000 7fcaa33f7640 1 -- start start 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c100e60 0x7fca9c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 0x7fca9c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c10ca00 con 0x7fca9c102040 2024-04-23T18:13:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c10c880 con 0x7fca9c102a00 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c10cb80 con 0x7fca9c100e60 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43438/0 (socket says 172.21.15.32:43438) 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 -- 172.21.15.32:0/3832488145 learned_addr learned my addr 172.21.15.32:0/3832488145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa1193640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c100e60 0x7fca9c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa1994640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 0x7fca9c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 -- 172.21.15.32:0/3832488145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c100e60 msgr2=0x7fca9c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c100e60 0x7fca9c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 -- 172.21.15.32:0/3832488145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 msgr2=0x7fca9c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 0x7fca9c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 -- 172.21.15.32:0/3832488145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c19fe20 con 0x7fca9c102040 2024-04-23T18:13:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa1994640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 0x7fca9c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa0992640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca9800e9f0 tx=0x7fca9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9800cd80 con 0x7fca9c102040 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca980054d0 con 0x7fca9c102040 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c1a0110 con 0x7fca9c102040 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98010690 con 0x7fca9c102040 2024-04-23T18:13:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.016+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c1a0610 con 0x7fca9c102040 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.018+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca9c1a08d0 con 0x7fca9c102040 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.019+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fca98004e90 con 0x7fca9c102040 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.019+0000 7fca927fc640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 0x7fca880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.019+0000 7fcaa1193640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 0x7fca880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.019+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fca980d0750 con 0x7fca9c102040 2024-04-23T18:13:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.020+0000 7fcaa1193640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 0x7fca880a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca9c101bb0 tx=0x7fca8c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.024+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca9809d280 con 0x7fca9c102040 2024-04-23T18:13:12.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.250+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca9c1a0aa0 con 0x7fca9c102040 2024-04-23T18:13:12.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.251+0000 7fca927fc640 1 -- 172.21.15.32:0/3832488145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fca9809cb80 con 0x7fca9c102040 2024-04-23T18:13:12.245 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:12.245 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:13:12.245 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:13:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 msgr2=0x7fca880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 0x7fca880a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca9c101bb0 tx=0x7fca8c00b390 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 msgr2=0x7fca9c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca9800e9f0 tx=0x7fca9800edf0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 shutdown_connections 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fca880a3bb0 0x7fca880a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c102a00 0x7fca9c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca9c102040 0x7fca9c198740 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 --2- 172.21.15.32:0/3832488145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca9c100e60 0x7fca9c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 >> 172.21.15.32:0/3832488145 conn(0x7fca9c0fc420 msgr2=0x7fca9c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 shutdown_connections 2024-04-23T18:13:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.253+0000 7fcaa33f7640 1 -- 172.21.15.32:0/3832488145 wait complete. 2024-04-23T18:13:12.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:12 smithi032 ceph-mon[29221]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:12 smithi032 ceph-mon[29221]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:12 smithi032 ceph-mon[29221]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:13:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 -- 172.21.15.32:0/3058190043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 msgr2=0x7fded010a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 --2- 172.21.15.32:0/3058190043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded010a460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009ca0 tx=0x7fdec802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 -- 172.21.15.32:0/3058190043 shutdown_connections 2024-04-23T18:13:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 --2- 172.21.15.32:0/3058190043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded010a460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 --2- 172.21.15.32:0/3058190043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 0x7fded0103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 --2- 172.21.15.32:0/3058190043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 0x7fded0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 -- 172.21.15.32:0/3058190043 >> 172.21.15.32:0/3058190043 conn(0x7fded00fd570 msgr2=0x7fded00ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 -- 172.21.15.32:0/3058190043 shutdown_connections 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.366+0000 7fded54c2640 1 -- 172.21.15.32:0/3058190043 wait complete. 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 Processor -- start 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 -- start start 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 0x7fded0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 0x7fded0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fded010daf0 con 0x7fded0103bd0 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fded010d970 con 0x7fded0102020 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.367+0000 7fded54c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fded010dc70 con 0x7fded0103210 2024-04-23T18:13:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43460/0 (socket says 172.21.15.32:43460) 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 -- 172.21.15.32:0/3189915733 learned_addr learned my addr 172.21.15.32:0/3189915733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdece7fc640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 0x7fded0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdeceffd640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 0x7fded0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 -- 172.21.15.32:0/3189915733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 msgr2=0x7fded0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 0x7fded0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 -- 172.21.15.32:0/3189915733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 msgr2=0x7fded0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 0x7fded0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 -- 172.21.15.32:0/3189915733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded019fd60 con 0x7fded0103bd0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdecf7fe640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdec80097b0 tx=0x7fdec8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8004240 con 0x7fded0103bd0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdec80317c0 con 0x7fded0103bd0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded019fff0 con 0x7fded0103bd0 2024-04-23T18:13:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec802fd80 con 0x7fded0103bd0 2024-04-23T18:13:12.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.368+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded01a04b0 con 0x7fded0103bd0 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.370+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded0109340 con 0x7fded0103bd0 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.370+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdec80043e0 con 0x7fded0103bd0 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.371+0000 7fdec7fff640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 0x7fdebc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.371+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdec80f1ba0 con 0x7fded0103bd0 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.371+0000 7fdeceffd640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 0x7fdebc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.372+0000 7fdeceffd640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 0x7fdebc0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdec0005d00 tx=0x7fdec0005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:12 smithi040 ceph-mon[32626]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:12 smithi040 ceph-mon[32626]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:12 smithi040 ceph-mon[32626]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.375+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdec80be7f0 con 0x7fded0103bd0 2024-04-23T18:13:12.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:12 smithi064 ceph-mon[33062]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:12 smithi064 ceph-mon[33062]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:12 smithi064 ceph-mon[33062]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.564+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fded002d770 con 0x7fded0103bd0 2024-04-23T18:13:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.565+0000 7fdec7fff640 1 -- 172.21.15.32:0/3189915733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22148 (secure 0 0 0) 0x7fdec80098b0 con 0x7fded0103bd0 2024-04-23T18:13:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:12.560 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:06.508628+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 msgr2=0x7fdebc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 0x7fdebc0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdec0005d00 tx=0x7fdec0005c50 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 msgr2=0x7fded019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdec80097b0 tx=0x7fdec8003c20 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 shutdown_connections 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fded0103bd0 0x7fded019f6f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fded0103210 0x7fded0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fdebc0a3bb0 0x7fdebc0a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 --2- 172.21.15.32:0/3189915733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0102020 0x7fded0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 >> 172.21.15.32:0/3189915733 conn(0x7fded00fd570 msgr2=0x7fded0106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.567+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 shutdown_connections 2024-04-23T18:13:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.568+0000 7fded54c2640 1 -- 172.21.15.32:0/3189915733 wait complete. 2024-04-23T18:13:12.573 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T18:13:12.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:13:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 -- 172.21.15.32:0/207034484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0100e60 msgr2=0x7f90a0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 --2- 172.21.15.32:0/207034484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0100e60 0x7f90a0101280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9098009900 tx=0x7f909802a410 comp rx=0 tx=0).stop 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 -- 172.21.15.32:0/207034484 shutdown_connections 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 --2- 172.21.15.32:0/207034484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102a00 0x7f90a0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 --2- 172.21.15.32:0/207034484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0102040 0x7f90a01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 --2- 172.21.15.32:0/207034484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0100e60 0x7f90a0101280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 -- 172.21.15.32:0/207034484 >> 172.21.15.32:0/207034484 conn(0x7f90a00fc460 msgr2=0x7f90a00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 -- 172.21.15.32:0/207034484 shutdown_connections 2024-04-23T18:13:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.681+0000 7f90a49fe640 1 -- 172.21.15.32:0/207034484 wait complete. 2024-04-23T18:13:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 Processor -- start 2024-04-23T18:13:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 -- start start 2024-04-23T18:13:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0100e60 0x7f90a0104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102040 0x7f90a01034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f90a010c9d0 con 0x7f90a0102a00 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90a010c850 con 0x7f90a0102040 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f90a49fe640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f90a010cb50 con 0x7f90a0100e60 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f909ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f909ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43478/0 (socket says 172.21.15.32:43478) 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f909ed9d640 1 -- 172.21.15.32:0/3123881157 learned_addr learned my addr 172.21.15.32:0/3123881157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f909e59c640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0100e60 0x7f90a0104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.682+0000 7f909dd9b640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102040 0x7f90a01034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 -- 172.21.15.32:0/3123881157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0100e60 msgr2=0x7f90a0104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0100e60 0x7f90a0104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 -- 172.21.15.32:0/3123881157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102040 msgr2=0x7f90a01034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102040 0x7f90a01034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 -- 172.21.15.32:0/3123881157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a0104280 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f909ed9d640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f909000eaa0 tx=0x7f909000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909000cdb0 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9090005020 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a01a2310 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90900116d0 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.683+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a01a2810 con 0x7f90a0102a00 2024-04-23T18:13:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.684+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a01a2b30 con 0x7f90a0102a00 2024-04-23T18:13:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.685+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f90900051d0 con 0x7f90a0102a00 2024-04-23T18:13:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.685+0000 7f908f7fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 0x7f90880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.686+0000 7f909e59c640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 0x7f90880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.686+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f90900d0440 con 0x7f90a0102a00 2024-04-23T18:13:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.686+0000 7f909e59c640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 0x7f90880a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9098009230 tx=0x7f90980314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.690+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9090016030 con 0x7f90a0102a00 2024-04-23T18:13:12.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.873+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90a006ad70 con 0x7f90880a3bb0 2024-04-23T18:13:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.882+0000 7f908f7fe640 1 -- 172.21.15.32:0/3123881157 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258889 (secure 0 0 0) 0x7f90a006ad70 con 0x7f90880a3bb0 2024-04-23T18:13:12.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:12.881 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 msgr2=0x7f90880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 0x7f90880a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9098009230 tx=0x7f90980314a0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 msgr2=0x7f90a01039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f909000eaa0 tx=0x7f909000ef60 comp rx=0 tx=0).stop 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 shutdown_connections 2024-04-23T18:13:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a0102a00 0x7f90a01039f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f90880a3bb0 0x7f90880a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a0102040 0x7f90a01034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 --2- 172.21.15.32:0/3123881157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90a0100e60 0x7f90a0104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 >> 172.21.15.32:0/3123881157 conn(0x7f90a00fc460 msgr2=0x7f90a006a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 shutdown_connections 2024-04-23T18:13:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:12.885+0000 7f90a49fe640 1 -- 172.21.15.32:0/3123881157 wait complete. 2024-04-23T18:13:12.890 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-23T18:13:12.066855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592980,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":261,"num_write_kb":2956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376992,"kb_used_data":11140,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564832,"statfs":{"total":1151940427776,"available":1151554387968,"internally_reserved":0,"allocated":11407360,"data_stored":7081166,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2612,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.063145"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256305+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:13:07.256305+0000","last_peered":"2024-04-23T18:13:07.256305+0000","last_clean":"2024-04-23T18:13:07.256305+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:13:07.256305+0000","last_undegraded":"2024-04-23T18:13:07.256305+0000","last_fullsized":"2024-04-23T18:13:07.256305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256227+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:13:07.256227+0000","last_peered":"2024-04-23T18:13:07.256227+0000","last_clean":"2024-04-23T18:13:07.256227+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:13:07.256227+0000","last_undegraded":"2024-04-23T18:13:07.256227+0000","last_fullsized":"2024-04-23T18:13:07.256227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256196+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:13:07.256196+0000","last_peered":"2024-04-23T18:13:07.256196+0000","last_clean":"2024-04-23T18:13:07.256196+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:13:07.256196+0000","last_undegraded":"2024-04-23T18:13:07.256196+0000","last_fullsized":"2024-04-23T18:13:07.256196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513659+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:13:06.513659+0000","last_peered":"2024-04-23T18:13:06.513659+0000","last_clean":"2024-04-23T18:13:06.513659+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:13:06.513659+0000","last_undegraded":"2024-04-23T18:13:06.513659+0000","last_fullsized":"2024-04-23T18:13:06.513659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315194+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:13:07.315194+0000","last_peered":"2024-04-23T18:13:07.315194+0000","last_clean":"2024-04-23T18:13:07.315194+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:13:07.315194+0000","last_undegraded":"2024-04-23T18:13:07.315194+0000","last_fullsized":"2024-04-23T18:13:07.315194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256372+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:13:07.256372+0000","last_peered":"2024-04-23T18:13:07.256372+0000","last_clean":"2024-04-23T18:13:07.256372+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:13:07.256372+0000","last_undegraded":"2024-04-23T18:13:07.256372+0000","last_fullsized":"2024-04-23T18:13:07.256372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256854+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:13:07.256854+0000","last_peered":"2024-04-23T18:13:07.256854+0000","last_clean":"2024-04-23T18:13:07.256854+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:13:07.256854+0000","last_undegraded":"2024-04-23T18:13:07.256854+0000","last_fullsized":"2024-04-23T18:13:07.256854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511606+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:13:06.511606+0000","last_peered":"2024-04-23T18:13:06.511606+0000","last_clean":"2024-04-23T18:13:06.511606+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:13:06.511606+0000","last_undegraded":"2024-04-23T18:13:06.511606+0000","last_fullsized":"2024-04-23T18:13:06.511606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664435+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:13:06.664435+0000","last_peered":"2024-04-23T18:13:06.664435+0000","last_clean":"2024-04-23T18:13:06.664435+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:13:06.664435+0000","last_undegraded":"2024-04-23T18:13:06.664435+0000","last_fullsized":"2024-04-23T18:13:06.664435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256612+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:13:07.256612+0000","last_peered":"2024-04-23T18:13:07.256612+0000","last_clean":"2024-04-23T18:13:07.256612+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:13:07.256612+0000","last_undegraded":"2024-04-23T18:13:07.256612+0000","last_fullsized":"2024-04-23T18:13:07.256612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256826+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:13:07.256826+0000","last_peered":"2024-04-23T18:13:07.256826+0000","last_clean":"2024-04-23T18:13:07.256826+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:13:07.256826+0000","last_undegraded":"2024-04-23T18:13:07.256826+0000","last_fullsized":"2024-04-23T18:13:07.256826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513661+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:13:06.513661+0000","last_peered":"2024-04-23T18:13:06.513661+0000","last_clean":"2024-04-23T18:13:06.513661+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:13:06.513661+0000","last_undegraded":"2024-04-23T18:13:06.513661+0000","last_fullsized":"2024-04-23T18:13:06.513661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256227+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:13:07.256227+0000","last_peered":"2024-04-23T18:13:07.256227+0000","last_clean":"2024-04-23T18:13:07.256227+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:13:07.256227+0000","last_undegraded":"2024-04-23T18:13:07.256227+0000","last_fullsized":"2024-04-23T18:13:07.256227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255744+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:13:07.255744+0000","last_peered":"2024-04-23T18:13:07.255744+0000","last_clean":"2024-04-23T18:13:07.255744+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:13:07.255744+0000","last_undegraded":"2024-04-23T18:13:07.255744+0000","last_fullsized":"2024-04-23T18:13:07.255744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256057+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:13:07.256057+0000","last_peered":"2024-04-23T18:13:07.256057+0000","last_clean":"2024-04-23T18:13:07.256057+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:13:07.256057+0000","last_undegraded":"2024-04-23T18:13:07.256057+0000","last_fullsized":"2024-04-23T18:13:07.256057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663903+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:13:06.663903+0000","last_peered":"2024-04-23T18:13:06.663903+0000","last_clean":"2024-04-23T18:13:06.663903+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:13:06.663903+0000","last_undegraded":"2024-04-23T18:13:06.663903+0000","last_fullsized":"2024-04-23T18:13:06.663903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256614+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:13:07.256614+0000","last_peered":"2024-04-23T18:13:07.256614+0000","last_clean":"2024-04-23T18:13:07.256614+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:13:07.256614+0000","last_undegraded":"2024-04-23T18:13:07.256614+0000","last_fullsized":"2024-04-23T18:13:07.256614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135521+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:13:07.135521+0000","last_peered":"2024-04-23T18:13:07.135521+0000","last_clean":"2024-04-23T18:13:07.135521+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:13:07.135521+0000","last_undegraded":"2024-04-23T18:13:07.135521+0000","last_fullsized":"2024-04-23T18:13:07.135521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256377+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:13:07.256377+0000","last_peered":"2024-04-23T18:13:07.256377+0000","last_clean":"2024-04-23T18:13:07.256377+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:13:07.256377+0000","last_undegraded":"2024-04-23T18:13:07.256377+0000","last_fullsized":"2024-04-23T18:13:07.256377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.050910+0000","last_change":"2024-04-23T18:13:05.030050+0000","last_active":"2024-04-23T18:13:09.050910+0000","last_peered":"2024-04-23T18:13:09.050910+0000","last_clean":"2024-04-23T18:13:09.050910+0000","last_became_active":"2024-04-23T18:13:05.029704+0000","last_became_peered":"2024-04-23T18:13:05.029704+0000","last_unstale":"2024-04-23T18:13:09.050910+0000","last_undegraded":"2024-04-23T18:13:09.050910+0000","last_fullsized":"2024-04-23T18:13:09.050910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:44.158663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511846+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:13:06.511846+0000","last_peered":"2024-04-23T18:13:06.511846+0000","last_clean":"2024-04-23T18:13:06.511846+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:13:06.511846+0000","last_undegraded":"2024-04-23T18:13:06.511846+0000","last_fullsized":"2024-04-23T18:13:06.511846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512190+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:13:06.512190+0000","last_peered":"2024-04-23T18:13:06.512190+0000","last_clean":"2024-04-23T18:13:06.512190+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:13:06.512190+0000","last_undegraded":"2024-04-23T18:13:06.512190+0000","last_fullsized":"2024-04-23T18:13:06.512190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256496+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:13:07.256496+0000","last_peered":"2024-04-23T18:13:07.256496+0000","last_clean":"2024-04-23T18:13:07.256496+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:13:07.256496+0000","last_undegraded":"2024-04-23T18:13:07.256496+0000","last_fullsized":"2024-04-23T18:13:07.256496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315431+0000","last_change":"2024-04-23T18:13:05.029770+0000","last_active":"2024-04-23T18:13:07.315431+0000","last_peered":"2024-04-23T18:13:07.315431+0000","last_clean":"2024-04-23T18:13:07.315431+0000","last_became_active":"2024-04-23T18:13:05.029270+0000","last_became_peered":"2024-04-23T18:13:05.029270+0000","last_unstale":"2024-04-23T18:13:07.315431+0000","last_undegraded":"2024-04-23T18:13:07.315431+0000","last_fullsized":"2024-04-23T18:13:07.315431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:12:56.065708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663985+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:13:06.663985+0000","last_peered":"2024-04-23T18:13:06.663985+0000","last_clean":"2024-04-23T18:13:06.663985+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:13:06.663985+0000","last_undegraded":"2024-04-23T18:13:06.663985+0000","last_fullsized":"2024-04-23T18:13:06.663985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315326+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:13:07.315326+0000","last_peered":"2024-04-23T18:13:07.315326+0000","last_clean":"2024-04-23T18:13:07.315326+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:13:07.315326+0000","last_undegraded":"2024-04-23T18:13:07.315326+0000","last_fullsized":"2024-04-23T18:13:07.315326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512846+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:13:06.512846+0000","last_peered":"2024-04-23T18:13:06.512846+0000","last_clean":"2024-04-23T18:13:06.512846+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:13:06.512846+0000","last_undegraded":"2024-04-23T18:13:06.512846+0000","last_fullsized":"2024-04-23T18:13:06.512846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512863+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:13:06.512863+0000","last_peered":"2024-04-23T18:13:06.512863+0000","last_clean":"2024-04-23T18:13:06.512863+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:13:06.512863+0000","last_undegraded":"2024-04-23T18:13:06.512863+0000","last_fullsized":"2024-04-23T18:13:06.512863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315259+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:13:07.315259+0000","last_peered":"2024-04-23T18:13:07.315259+0000","last_clean":"2024-04-23T18:13:07.315259+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:13:07.315259+0000","last_undegraded":"2024-04-23T18:13:07.315259+0000","last_fullsized":"2024-04-23T18:13:07.315259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256754+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:13:07.256754+0000","last_peered":"2024-04-23T18:13:07.256754+0000","last_clean":"2024-04-23T18:13:07.256754+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:13:07.256754+0000","last_undegraded":"2024-04-23T18:13:07.256754+0000","last_fullsized":"2024-04-23T18:13:07.256754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513015+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:13:06.513015+0000","last_peered":"2024-04-23T18:13:06.513015+0000","last_clean":"2024-04-23T18:13:06.513015+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:13:06.513015+0000","last_undegraded":"2024-04-23T18:13:06.513015+0000","last_fullsized":"2024-04-23T18:13:06.513015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255690+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:13:07.255690+0000","last_peered":"2024-04-23T18:13:07.255690+0000","last_clean":"2024-04-23T18:13:07.255690+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:13:07.255690+0000","last_undegraded":"2024-04-23T18:13:07.255690+0000","last_fullsized":"2024-04-23T18:13:07.255690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511541+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:13:06.511541+0000","last_peered":"2024-04-23T18:13:06.511541+0000","last_clean":"2024-04-23T18:13:06.511541+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:13:06.511541+0000","last_undegraded":"2024-04-23T18:13:06.511541+0000","last_fullsized":"2024-04-23T18:13:06.511541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512418+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:13:06.512418+0000","last_peered":"2024-04-23T18:13:06.512418+0000","last_clean":"2024-04-23T18:13:06.512418+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:13:06.512418+0000","last_undegraded":"2024-04-23T18:13:06.512418+0000","last_fullsized":"2024-04-23T18:13:06.512418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135150+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:13:07.135150+0000","last_peered":"2024-04-23T18:13:07.135150+0000","last_clean":"2024-04-23T18:13:07.135150+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:13:07.135150+0000","last_undegraded":"2024-04-23T18:13:07.135150+0000","last_fullsized":"2024-04-23T18:13:07.135150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039130+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:13:08.039130+0000","last_peered":"2024-04-23T18:13:08.039130+0000","last_clean":"2024-04-23T18:13:08.039130+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:13:08.039130+0000","last_undegraded":"2024-04-23T18:13:08.039130+0000","last_fullsized":"2024-04-23T18:13:08.039130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664465+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:13:06.664465+0000","last_peered":"2024-04-23T18:13:06.664465+0000","last_clean":"2024-04-23T18:13:06.664465+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:13:06.664465+0000","last_undegraded":"2024-04-23T18:13:06.664465+0000","last_fullsized":"2024-04-23T18:13:06.664465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039850+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:13:08.039850+0000","last_peered":"2024-04-23T18:13:08.039850+0000","last_clean":"2024-04-23T18:13:08.039850+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:13:08.039850+0000","last_undegraded":"2024-04-23T18:13:08.039850+0000","last_fullsized":"2024-04-23T18:13:08.039850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256401+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:13:07.256401+0000","last_peered":"2024-04-23T18:13:07.256401+0000","last_clean":"2024-04-23T18:13:07.256401+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:13:07.256401+0000","last_undegraded":"2024-04-23T18:13:07.256401+0000","last_fullsized":"2024-04-23T18:13:07.256401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513008+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:13:06.513008+0000","last_peered":"2024-04-23T18:13:06.513008+0000","last_clean":"2024-04-23T18:13:06.513008+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:13:06.513008+0000","last_undegraded":"2024-04-23T18:13:06.513008+0000","last_fullsized":"2024-04-23T18:13:06.513008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664283+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:06.664283+0000","last_peered":"2024-04-23T18:13:06.664283+0000","last_clean":"2024-04-23T18:13:06.664283+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:13:06.664283+0000","last_undegraded":"2024-04-23T18:13:06.664283+0000","last_fullsized":"2024-04-23T18:13:06.664283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.038123+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:13:08.038123+0000","last_peered":"2024-04-23T18:13:08.038123+0000","last_clean":"2024-04-23T18:13:08.038123+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:13:08.038123+0000","last_undegraded":"2024-04-23T18:13:08.038123+0000","last_fullsized":"2024-04-23T18:13:08.038123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315311+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:13:07.315311+0000","last_peered":"2024-04-23T18:13:07.315311+0000","last_clean":"2024-04-23T18:13:07.315311+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:13:07.315311+0000","last_undegraded":"2024-04-23T18:13:07.315311+0000","last_fullsized":"2024-04-23T18:13:07.315311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255834+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:13:07.255834+0000","last_peered":"2024-04-23T18:13:07.255834+0000","last_clean":"2024-04-23T18:13:07.255834+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:13:07.255834+0000","last_undegraded":"2024-04-23T18:13:07.255834+0000","last_fullsized":"2024-04-23T18:13:07.255834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255679+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:13:07.255679+0000","last_peered":"2024-04-23T18:13:07.255679+0000","last_clean":"2024-04-23T18:13:07.255679+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:13:07.255679+0000","last_undegraded":"2024-04-23T18:13:07.255679+0000","last_fullsized":"2024-04-23T18:13:07.255679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512635+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:13:06.512635+0000","last_peered":"2024-04-23T18:13:06.512635+0000","last_clean":"2024-04-23T18:13:06.512635+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:13:06.512635+0000","last_undegraded":"2024-04-23T18:13:06.512635+0000","last_fullsized":"2024-04-23T18:13:06.512635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256616+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:07.256616+0000","last_peered":"2024-04-23T18:13:07.256616+0000","last_clean":"2024-04-23T18:13:07.256616+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:13:07.256616+0000","last_undegraded":"2024-04-23T18:13:07.256616+0000","last_fullsized":"2024-04-23T18:13:07.256616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513322+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:13:06.513322+0000","last_peered":"2024-04-23T18:13:06.513322+0000","last_clean":"2024-04-23T18:13:06.513322+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:13:06.513322+0000","last_undegraded":"2024-04-23T18:13:06.513322+0000","last_fullsized":"2024-04-23T18:13:06.513322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256999+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:13:07.256999+0000","last_peered":"2024-04-23T18:13:07.256999+0000","last_clean":"2024-04-23T18:13:07.256999+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:13:07.256999+0000","last_undegraded":"2024-04-23T18:13:07.256999+0000","last_fullsized":"2024-04-23T18:13:07.256999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.038467+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:13:08.038467+0000","last_peered":"2024-04-23T18:13:08.038467+0000","last_clean":"2024-04-23T18:13:08.038467+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:13:08.038467+0000","last_undegraded":"2024-04-23T18:13:08.038467+0000","last_fullsized":"2024-04-23T18:13:08.038467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513039+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:06.513039+0000","last_peered":"2024-04-23T18:13:06.513039+0000","last_clean":"2024-04-23T18:13:06.513039+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:13:06.513039+0000","last_undegraded":"2024-04-23T18:13:06.513039+0000","last_fullsized":"2024-04-23T18:13:06.513039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315463+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:13:07.315463+0000","last_peered":"2024-04-23T18:13:07.315463+0000","last_clean":"2024-04-23T18:13:07.315463+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:13:07.315463+0000","last_undegraded":"2024-04-23T18:13:07.315463+0000","last_fullsized":"2024-04-23T18:13:07.315463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256403+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:07.256403+0000","last_peered":"2024-04-23T18:13:07.256403+0000","last_clean":"2024-04-23T18:13:07.256403+0000","last_became_active":"2024-04-23T18:13:05.534622+0000","last_became_peered":"2024-04-23T18:13:05.534622+0000","last_unstale":"2024-04-23T18:13:07.256403+0000","last_undegraded":"2024-04-23T18:13:07.256403+0000","last_fullsized":"2024-04-23T18:13:07.256403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:45:59.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135310+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:13:07.135310+0000","last_peered":"2024-04-23T18:13:07.135310+0000","last_clean":"2024-04-23T18:13:07.135310+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:13:07.135310+0000","last_undegraded":"2024-04-23T18:13:07.135310+0000","last_fullsized":"2024-04-23T18:13:07.135310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256675+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:13:07.256675+0000","last_peered":"2024-04-23T18:13:07.256675+0000","last_clean":"2024-04-23T18:13:07.256675+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:13:07.256675+0000","last_undegraded":"2024-04-23T18:13:07.256675+0000","last_fullsized":"2024-04-23T18:13:07.256675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257087+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:13:07.257087+0000","last_peered":"2024-04-23T18:13:07.257087+0000","last_clean":"2024-04-23T18:13:07.257087+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:13:07.257087+0000","last_undegraded":"2024-04-23T18:13:07.257087+0000","last_fullsized":"2024-04-23T18:13:07.257087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257291+0000","last_change":"2024-04-23T18:13:05.532689+0000","last_active":"2024-04-23T18:13:07.257291+0000","last_peered":"2024-04-23T18:13:07.257291+0000","last_clean":"2024-04-23T18:13:07.257291+0000","last_became_active":"2024-04-23T18:13:05.532383+0000","last_became_peered":"2024-04-23T18:13:05.532383+0000","last_unstale":"2024-04-23T18:13:07.257291+0000","last_undegraded":"2024-04-23T18:13:07.257291+0000","last_fullsized":"2024-04-23T18:13:07.257291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:42.869965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512650+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:13:06.512650+0000","last_peered":"2024-04-23T18:13:06.512650+0000","last_clean":"2024-04-23T18:13:06.512650+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:13:06.512650+0000","last_undegraded":"2024-04-23T18:13:06.512650+0000","last_fullsized":"2024-04-23T18:13:06.512650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255860+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:13:07.255860+0000","last_peered":"2024-04-23T18:13:07.255860+0000","last_clean":"2024-04-23T18:13:07.255860+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:13:07.255860+0000","last_undegraded":"2024-04-23T18:13:07.255860+0000","last_fullsized":"2024-04-23T18:13:07.255860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511964+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:13:06.511964+0000","last_peered":"2024-04-23T18:13:06.511964+0000","last_clean":"2024-04-23T18:13:06.511964+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:13:06.511964+0000","last_undegraded":"2024-04-23T18:13:06.511964+0000","last_fullsized":"2024-04-23T18:13:06.511964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513580+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:13:06.513580+0000","last_peered":"2024-04-23T18:13:06.513580+0000","last_clean":"2024-04-23T18:13:06.513580+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:13:06.513580+0000","last_undegraded":"2024-04-23T18:13:06.513580+0000","last_fullsized":"2024-04-23T18:13:06.513580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":351,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256355+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:13:07.256355+0000","last_peered":"2024-04-23T18:13:07.256355+0000","last_clean":"2024-04-23T18:13:07.256355+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:13:07.256355+0000","last_undegraded":"2024-04-23T18:13:07.256355+0000","last_fullsized":"2024-04-23T18:13:07.256355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255536+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:13:07.255536+0000","last_peered":"2024-04-23T18:13:07.255536+0000","last_clean":"2024-04-23T18:13:07.255536+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:13:07.255536+0000","last_undegraded":"2024-04-23T18:13:07.255536+0000","last_fullsized":"2024-04-23T18:13:07.255536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.047857+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:13:09.047857+0000","last_peered":"2024-04-23T18:13:09.047857+0000","last_clean":"2024-04-23T18:13:09.047857+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:13:09.047857+0000","last_undegraded":"2024-04-23T18:13:09.047857+0000","last_fullsized":"2024-04-23T18:13:09.047857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.037646+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:13:08.037646+0000","last_peered":"2024-04-23T18:13:08.037646+0000","last_clean":"2024-04-23T18:13:08.037646+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:13:08.037646+0000","last_undegraded":"2024-04-23T18:13:08.037646+0000","last_fullsized":"2024-04-23T18:13:08.037646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255836+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:13:07.255836+0000","last_peered":"2024-04-23T18:13:07.255836+0000","last_clean":"2024-04-23T18:13:07.255836+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:13:07.255836+0000","last_undegraded":"2024-04-23T18:13:07.255836+0000","last_fullsized":"2024-04-23T18:13:07.255836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.049776+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:13:09.049776+0000","last_peered":"2024-04-23T18:13:09.049776+0000","last_clean":"2024-04-23T18:13:09.049776+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:13:09.049776+0000","last_undegraded":"2024-04-23T18:13:09.049776+0000","last_fullsized":"2024-04-23T18:13:09.049776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511816+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:13:06.511816+0000","last_peered":"2024-04-23T18:13:06.511816+0000","last_clean":"2024-04-23T18:13:06.511816+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:13:06.511816+0000","last_undegraded":"2024-04-23T18:13:06.511816+0000","last_fullsized":"2024-04-23T18:13:06.511816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513650+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:13:06.513650+0000","last_peered":"2024-04-23T18:13:06.513650+0000","last_clean":"2024-04-23T18:13:06.513650+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:13:06.513650+0000","last_undegraded":"2024-04-23T18:13:06.513650+0000","last_fullsized":"2024-04-23T18:13:06.513650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256388+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:13:07.256388+0000","last_peered":"2024-04-23T18:13:07.256388+0000","last_clean":"2024-04-23T18:13:07.256388+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:13:07.256388+0000","last_undegraded":"2024-04-23T18:13:07.256388+0000","last_fullsized":"2024-04-23T18:13:07.256388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663869+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:13:06.663869+0000","last_peered":"2024-04-23T18:13:06.663869+0000","last_clean":"2024-04-23T18:13:06.663869+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:13:06.663869+0000","last_undegraded":"2024-04-23T18:13:06.663869+0000","last_fullsized":"2024-04-23T18:13:06.663869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315053+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:13:07.315053+0000","last_peered":"2024-04-23T18:13:07.315053+0000","last_clean":"2024-04-23T18:13:07.315053+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:13:07.315053+0000","last_undegraded":"2024-04-23T18:13:07.315053+0000","last_fullsized":"2024-04-23T18:13:07.315053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.040686+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:13:08.040686+0000","last_peered":"2024-04-23T18:13:08.040686+0000","last_clean":"2024-04-23T18:13:08.040686+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:13:08.040686+0000","last_undegraded":"2024-04-23T18:13:08.040686+0000","last_fullsized":"2024-04-23T18:13:08.040686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255864+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:13:07.255864+0000","last_peered":"2024-04-23T18:13:07.255864+0000","last_clean":"2024-04-23T18:13:07.255864+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:13:07.255864+0000","last_undegraded":"2024-04-23T18:13:07.255864+0000","last_fullsized":"2024-04-23T18:13:07.255864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256399+0000","last_change":"2024-04-23T18:13:05.023543+0000","last_active":"2024-04-23T18:13:07.256399+0000","last_peered":"2024-04-23T18:13:07.256399+0000","last_clean":"2024-04-23T18:13:07.256399+0000","last_became_active":"2024-04-23T18:13:05.023066+0000","last_became_peered":"2024-04-23T18:13:05.023066+0000","last_unstale":"2024-04-23T18:13:07.256399+0000","last_undegraded":"2024-04-23T18:13:07.256399+0000","last_fullsized":"2024-04-23T18:13:07.256399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:09.234277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512015+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:13:06.512015+0000","last_peered":"2024-04-23T18:13:06.512015+0000","last_clean":"2024-04-23T18:13:06.512015+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:13:06.512015+0000","last_undegraded":"2024-04-23T18:13:06.512015+0000","last_fullsized":"2024-04-23T18:13:06.512015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039831+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:13:08.039831+0000","last_peered":"2024-04-23T18:13:08.039831+0000","last_clean":"2024-04-23T18:13:08.039831+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:13:08.039831+0000","last_undegraded":"2024-04-23T18:13:08.039831+0000","last_fullsized":"2024-04-23T18:13:08.039831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257245+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:13:07.257245+0000","last_peered":"2024-04-23T18:13:07.257245+0000","last_clean":"2024-04-23T18:13:07.257245+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:13:07.257245+0000","last_undegraded":"2024-04-23T18:13:07.257245+0000","last_fullsized":"2024-04-23T18:13:07.257245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256026+0000","last_change":"2024-04-23T18:13:05.020176+0000","last_active":"2024-04-23T18:13:07.256026+0000","last_peered":"2024-04-23T18:13:07.256026+0000","last_clean":"2024-04-23T18:13:07.256026+0000","last_became_active":"2024-04-23T18:13:05.019646+0000","last_became_peered":"2024-04-23T18:13:05.019646+0000","last_unstale":"2024-04-23T18:13:07.256026+0000","last_undegraded":"2024-04-23T18:13:07.256026+0000","last_fullsized":"2024-04-23T18:13:07.256026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.296751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256971+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:13:07.256971+0000","last_peered":"2024-04-23T18:13:07.256971+0000","last_clean":"2024-04-23T18:13:07.256971+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:13:07.256971+0000","last_undegraded":"2024-04-23T18:13:07.256971+0000","last_fullsized":"2024-04-23T18:13:07.256971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512663+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:13:06.512663+0000","last_peered":"2024-04-23T18:13:06.512663+0000","last_clean":"2024-04-23T18:13:06.512663+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:13:06.512663+0000","last_undegraded":"2024-04-23T18:13:06.512663+0000","last_fullsized":"2024-04-23T18:13:06.512663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257096+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:13:07.257096+0000","last_peered":"2024-04-23T18:13:07.257096+0000","last_clean":"2024-04-23T18:13:07.257096+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:13:07.257096+0000","last_undegraded":"2024-04-23T18:13:07.257096+0000","last_fullsized":"2024-04-23T18:13:07.257096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255968+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:13:07.255968+0000","last_peered":"2024-04-23T18:13:07.255968+0000","last_clean":"2024-04-23T18:13:07.255968+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:13:07.255968+0000","last_undegraded":"2024-04-23T18:13:07.255968+0000","last_fullsized":"2024-04-23T18:13:07.255968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256897+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:13:07.256897+0000","last_peered":"2024-04-23T18:13:07.256897+0000","last_clean":"2024-04-23T18:13:07.256897+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:13:07.256897+0000","last_undegraded":"2024-04-23T18:13:07.256897+0000","last_fullsized":"2024-04-23T18:13:07.256897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255948+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:13:07.255948+0000","last_peered":"2024-04-23T18:13:07.255948+0000","last_clean":"2024-04-23T18:13:07.255948+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:13:07.255948+0000","last_undegraded":"2024-04-23T18:13:07.255948+0000","last_fullsized":"2024-04-23T18:13:07.255948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255841+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:13:07.255841+0000","last_peered":"2024-04-23T18:13:07.255841+0000","last_clean":"2024-04-23T18:13:07.255841+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:13:07.255841+0000","last_undegraded":"2024-04-23T18:13:07.255841+0000","last_fullsized":"2024-04-23T18:13:07.255841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256466+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:13:07.256466+0000","last_peered":"2024-04-23T18:13:07.256466+0000","last_clean":"2024-04-23T18:13:07.256466+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:13:07.256466+0000","last_undegraded":"2024-04-23T18:13:07.256466+0000","last_fullsized":"2024-04-23T18:13:07.256466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256006+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:13:07.256006+0000","last_peered":"2024-04-23T18:13:07.256006+0000","last_clean":"2024-04-23T18:13:07.256006+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:13:07.256006+0000","last_undegraded":"2024-04-23T18:13:07.256006+0000","last_fullsized":"2024-04-23T18:13:07.256006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255671+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:13:07.255671+0000","last_peered":"2024-04-23T18:13:07.255671+0000","last_clean":"2024-04-23T18:13:07.255671+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:13:07.255671+0000","last_undegraded":"2024-04-23T18:13:07.255671+0000","last_fullsized":"2024-04-23T18:13:07.255671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256223+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:13:07.256223+0000","last_peered":"2024-04-23T18:13:07.256223+0000","last_clean":"2024-04-23T18:13:07.256223+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:13:07.256223+0000","last_undegraded":"2024-04-23T18:13:07.256223+0000","last_fullsized":"2024-04-23T18:13:07.256223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256148+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:13:07.256148+0000","last_peered":"2024-04-23T18:13:07.256148+0000","last_clean":"2024-04-23T18:13:07.256148+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:13:07.256148+0000","last_undegraded":"2024-04-23T18:13:07.256148+0000","last_fullsized":"2024-04-23T18:13:07.256148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511804+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:13:06.511804+0000","last_peered":"2024-04-23T18:13:06.511804+0000","last_clean":"2024-04-23T18:13:06.511804+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:13:06.511804+0000","last_undegraded":"2024-04-23T18:13:06.511804+0000","last_fullsized":"2024-04-23T18:13:06.511804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512527+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:13:06.512527+0000","last_peered":"2024-04-23T18:13:06.512527+0000","last_clean":"2024-04-23T18:13:06.512527+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:13:06.512527+0000","last_undegraded":"2024-04-23T18:13:06.512527+0000","last_fullsized":"2024-04-23T18:13:06.512527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511570+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:13:06.511570+0000","last_peered":"2024-04-23T18:13:06.511570+0000","last_clean":"2024-04-23T18:13:06.511570+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:13:06.511570+0000","last_undegraded":"2024-04-23T18:13:06.511570+0000","last_fullsized":"2024-04-23T18:13:06.511570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664082+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:13:06.664082+0000","last_peered":"2024-04-23T18:13:06.664082+0000","last_clean":"2024-04-23T18:13:06.664082+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:13:06.664082+0000","last_undegraded":"2024-04-23T18:13:06.664082+0000","last_fullsized":"2024-04-23T18:13:06.664082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256078+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:13:07.256078+0000","last_peered":"2024-04-23T18:13:07.256078+0000","last_clean":"2024-04-23T18:13:07.256078+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:13:07.256078+0000","last_undegraded":"2024-04-23T18:13:07.256078+0000","last_fullsized":"2024-04-23T18:13:07.256078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256133+0000","last_change":"2024-04-23T18:13:05.536877+0000","last_active":"2024-04-23T18:13:07.256133+0000","last_peered":"2024-04-23T18:13:07.256133+0000","last_clean":"2024-04-23T18:13:07.256133+0000","last_became_active":"2024-04-23T18:13:05.534611+0000","last_became_peered":"2024-04-23T18:13:05.534611+0000","last_unstale":"2024-04-23T18:13:07.256133+0000","last_undegraded":"2024-04-23T18:13:07.256133+0000","last_fullsized":"2024-04-23T18:13:07.256133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:55.986092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249130,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2612,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":443597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":1035951,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":443833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":937984,"data_stored":443641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1034587,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":925696,"data_stored":443015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36268,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708884,"statfs":{"total":95995035648,"available":95957897216,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":929792,"data_stored":436091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":443299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":434467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1033565,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:13:12.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:13:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 -- 172.21.15.32:0/1932773516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 msgr2=0x7fd4341023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 --2- 172.21.15.32:0/1932773516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd4341023d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd424009900 tx=0x7fd42402a3f0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 -- 172.21.15.32:0/1932773516 shutdown_connections 2024-04-23T18:13:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 --2- 172.21.15.32:0/1932773516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd434103b60 0x7fd43410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 --2- 172.21.15.32:0/1932773516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4341031a0 0x7fd434103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 --2- 172.21.15.32:0/1932773516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd4341023d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 -- 172.21.15.32:0/1932773516 >> 172.21.15.32:0/1932773516 conn(0x7fd4340fd540 msgr2=0x7fd4340ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 -- 172.21.15.32:0/1932773516 shutdown_connections 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.039+0000 7fd438afa640 1 -- 172.21.15.32:0/1932773516 wait complete. 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 Processor -- start 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 -- start start 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4341031a0 0x7fd434074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd434103b60 0x7fd434074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410dab0 con 0x7fd434101fb0 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410d930 con 0x7fd434103b60 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd438afa640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410dc30 con 0x7fd4341031a0 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43490/0 (socket says 172.21.15.32:43490) 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 -- 172.21.15.32:0/3144641962 learned_addr learned my addr 172.21.15.32:0/3144641962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 -- 172.21.15.32:0/3144641962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4341031a0 msgr2=0x7fd434074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4341031a0 0x7fd434074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 -- 172.21.15.32:0/3144641962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd434103b60 msgr2=0x7fd434074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd434103b60 0x7fd434074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.040+0000 7fd43259c640 1 -- 172.21.15.32:0/3144641962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd434075110 con 0x7fd434101fb0 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd43259c640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd424006010 tx=0x7fd424004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42403f030 con 0x7fd434101fb0 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd424003cb0 con 0x7fd434101fb0 2024-04-23T18:13:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4341a21b0 con 0x7fd434101fb0 2024-04-23T18:13:13.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424035070 con 0x7fd434101fb0 2024-04-23T18:13:13.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.041+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4341a2750 con 0x7fd434101fb0 2024-04-23T18:13:13.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.042+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4341092d0 con 0x7fd434101fb0 2024-04-23T18:13:13.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.043+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd424042030 con 0x7fd434101fb0 2024-04-23T18:13:13.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.043+0000 7fd4237fe640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 0x7fd41c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:13.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.044+0000 7fd431d9b640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 0x7fd41c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:13.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.044+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd4240f09f0 con 0x7fd434101fb0 2024-04-23T18:13:13.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.044+0000 7fd431d9b640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 0x7fd41c0a60f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd434075a70 tx=0x7fd428005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:13.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.048+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd424003e60 con 0x7fd434101fb0 2024-04-23T18:13:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.063+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd4240b9490 con 0x7fd434101fb0 2024-04-23T18:13:13.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.236+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd434062390 con 0x7fd41c0a3c30 2024-04-23T18:13:13.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.244+0000 7fd4237fe640 1 -- 172.21.15.32:0/3144641962 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258889 (secure 0 0 0) 0x7fd434062390 con 0x7fd41c0a3c30 2024-04-23T18:13:13.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:13:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 msgr2=0x7fd41c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 0x7fd41c0a60f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd434075a70 tx=0x7fd428005c70 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 msgr2=0x7fd434075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd424006010 tx=0x7fd424004430 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 shutdown_connections 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd41c0a3c30 0x7fd41c0a60f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd434103b60 0x7fd434074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4341031a0 0x7fd434074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 --2- 172.21.15.32:0/3144641962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd434101fb0 0x7fd434075d00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 >> 172.21.15.32:0/3144641962 conn(0x7fd4340fd540 msgr2=0x7fd4341064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 shutdown_connections 2024-04-23T18:13:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:13.247+0000 7fd438afa640 1 -- 172.21.15.32:0/3144641962 wait complete. 2024-04-23T18:13:13.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:13 smithi032 ceph-mon[29221]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s rd, 2.5 KiB/s wr, 11 op/s 2024-04-23T18:13:13.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:13 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3832488145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:13 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3189915733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:13 smithi032 ceph-mon[29221]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:13.254 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-23T18:13:12.066855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592980,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":261,"num_write_kb":2956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376992,"kb_used_data":11140,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564832,"statfs":{"total":1151940427776,"available":1151554387968,"internally_reserved":0,"allocated":11407360,"data_stored":7081166,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2612,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.063145"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256305+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:13:07.256305+0000","last_peered":"2024-04-23T18:13:07.256305+0000","last_clean":"2024-04-23T18:13:07.256305+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:13:07.256305+0000","last_undegraded":"2024-04-23T18:13:07.256305+0000","last_fullsized":"2024-04-23T18:13:07.256305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256227+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:13:07.256227+0000","last_peered":"2024-04-23T18:13:07.256227+0000","last_clean":"2024-04-23T18:13:07.256227+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:13:07.256227+0000","last_undegraded":"2024-04-23T18:13:07.256227+0000","last_fullsized":"2024-04-23T18:13:07.256227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256196+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:13:07.256196+0000","last_peered":"2024-04-23T18:13:07.256196+0000","last_clean":"2024-04-23T18:13:07.256196+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:13:07.256196+0000","last_undegraded":"2024-04-23T18:13:07.256196+0000","last_fullsized":"2024-04-23T18:13:07.256196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513659+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:13:06.513659+0000","last_peered":"2024-04-23T18:13:06.513659+0000","last_clean":"2024-04-23T18:13:06.513659+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:13:06.513659+0000","last_undegraded":"2024-04-23T18:13:06.513659+0000","last_fullsized":"2024-04-23T18:13:06.513659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315194+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:13:07.315194+0000","last_peered":"2024-04-23T18:13:07.315194+0000","last_clean":"2024-04-23T18:13:07.315194+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:13:07.315194+0000","last_undegraded":"2024-04-23T18:13:07.315194+0000","last_fullsized":"2024-04-23T18:13:07.315194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256372+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:13:07.256372+0000","last_peered":"2024-04-23T18:13:07.256372+0000","last_clean":"2024-04-23T18:13:07.256372+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:13:07.256372+0000","last_undegraded":"2024-04-23T18:13:07.256372+0000","last_fullsized":"2024-04-23T18:13:07.256372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256854+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:13:07.256854+0000","last_peered":"2024-04-23T18:13:07.256854+0000","last_clean":"2024-04-23T18:13:07.256854+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:13:07.256854+0000","last_undegraded":"2024-04-23T18:13:07.256854+0000","last_fullsized":"2024-04-23T18:13:07.256854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511606+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:13:06.511606+0000","last_peered":"2024-04-23T18:13:06.511606+0000","last_clean":"2024-04-23T18:13:06.511606+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:13:06.511606+0000","last_undegraded":"2024-04-23T18:13:06.511606+0000","last_fullsized":"2024-04-23T18:13:06.511606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664435+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:13:06.664435+0000","last_peered":"2024-04-23T18:13:06.664435+0000","last_clean":"2024-04-23T18:13:06.664435+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:13:06.664435+0000","last_undegraded":"2024-04-23T18:13:06.664435+0000","last_fullsized":"2024-04-23T18:13:06.664435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256612+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:13:07.256612+0000","last_peered":"2024-04-23T18:13:07.256612+0000","last_clean":"2024-04-23T18:13:07.256612+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:13:07.256612+0000","last_undegraded":"2024-04-23T18:13:07.256612+0000","last_fullsized":"2024-04-23T18:13:07.256612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256826+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:13:07.256826+0000","last_peered":"2024-04-23T18:13:07.256826+0000","last_clean":"2024-04-23T18:13:07.256826+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:13:07.256826+0000","last_undegraded":"2024-04-23T18:13:07.256826+0000","last_fullsized":"2024-04-23T18:13:07.256826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513661+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:13:06.513661+0000","last_peered":"2024-04-23T18:13:06.513661+0000","last_clean":"2024-04-23T18:13:06.513661+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:13:06.513661+0000","last_undegraded":"2024-04-23T18:13:06.513661+0000","last_fullsized":"2024-04-23T18:13:06.513661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256227+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:13:07.256227+0000","last_peered":"2024-04-23T18:13:07.256227+0000","last_clean":"2024-04-23T18:13:07.256227+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:13:07.256227+0000","last_undegraded":"2024-04-23T18:13:07.256227+0000","last_fullsized":"2024-04-23T18:13:07.256227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255744+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:13:07.255744+0000","last_peered":"2024-04-23T18:13:07.255744+0000","last_clean":"2024-04-23T18:13:07.255744+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:13:07.255744+0000","last_undegraded":"2024-04-23T18:13:07.255744+0000","last_fullsized":"2024-04-23T18:13:07.255744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256057+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:13:07.256057+0000","last_peered":"2024-04-23T18:13:07.256057+0000","last_clean":"2024-04-23T18:13:07.256057+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:13:07.256057+0000","last_undegraded":"2024-04-23T18:13:07.256057+0000","last_fullsized":"2024-04-23T18:13:07.256057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663903+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:13:06.663903+0000","last_peered":"2024-04-23T18:13:06.663903+0000","last_clean":"2024-04-23T18:13:06.663903+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:13:06.663903+0000","last_undegraded":"2024-04-23T18:13:06.663903+0000","last_fullsized":"2024-04-23T18:13:06.663903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256614+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:13:07.256614+0000","last_peered":"2024-04-23T18:13:07.256614+0000","last_clean":"2024-04-23T18:13:07.256614+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:13:07.256614+0000","last_undegraded":"2024-04-23T18:13:07.256614+0000","last_fullsized":"2024-04-23T18:13:07.256614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135521+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:13:07.135521+0000","last_peered":"2024-04-23T18:13:07.135521+0000","last_clean":"2024-04-23T18:13:07.135521+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:13:07.135521+0000","last_undegraded":"2024-04-23T18:13:07.135521+0000","last_fullsized":"2024-04-23T18:13:07.135521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256377+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:13:07.256377+0000","last_peered":"2024-04-23T18:13:07.256377+0000","last_clean":"2024-04-23T18:13:07.256377+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:13:07.256377+0000","last_undegraded":"2024-04-23T18:13:07.256377+0000","last_fullsized":"2024-04-23T18:13:07.256377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.050910+0000","last_change":"2024-04-23T18:13:05.030050+0000","last_active":"2024-04-23T18:13:09.050910+0000","last_peered":"2024-04-23T18:13:09.050910+0000","last_clean":"2024-04-23T18:13:09.050910+0000","last_became_active":"2024-04-23T18:13:05.029704+0000","last_became_peered":"2024-04-23T18:13:05.029704+0000","last_unstale":"2024-04-23T18:13:09.050910+0000","last_undegraded":"2024-04-23T18:13:09.050910+0000","last_fullsized":"2024-04-23T18:13:09.050910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:44.158663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511846+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:13:06.511846+0000","last_peered":"2024-04-23T18:13:06.511846+0000","last_clean":"2024-04-23T18:13:06.511846+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:13:06.511846+0000","last_undegraded":"2024-04-23T18:13:06.511846+0000","last_fullsized":"2024-04-23T18:13:06.511846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512190+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:13:06.512190+0000","last_peered":"2024-04-23T18:13:06.512190+0000","last_clean":"2024-04-23T18:13:06.512190+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:13:06.512190+0000","last_undegraded":"2024-04-23T18:13:06.512190+0000","last_fullsized":"2024-04-23T18:13:06.512190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256496+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:13:07.256496+0000","last_peered":"2024-04-23T18:13:07.256496+0000","last_clean":"2024-04-23T18:13:07.256496+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:13:07.256496+0000","last_undegraded":"2024-04-23T18:13:07.256496+0000","last_fullsized":"2024-04-23T18:13:07.256496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315431+0000","last_change":"2024-04-23T18:13:05.029770+0000","last_active":"2024-04-23T18:13:07.315431+0000","last_peered":"2024-04-23T18:13:07.315431+0000","last_clean":"2024-04-23T18:13:07.315431+0000","last_became_active":"2024-04-23T18:13:05.029270+0000","last_became_peered":"2024-04-23T18:13:05.029270+0000","last_unstale":"2024-04-23T18:13:07.315431+0000","last_undegraded":"2024-04-23T18:13:07.315431+0000","last_fullsized":"2024-04-23T18:13:07.315431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:12:56.065708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663985+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:13:06.663985+0000","last_peered":"2024-04-23T18:13:06.663985+0000","last_clean":"2024-04-23T18:13:06.663985+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:13:06.663985+0000","last_undegraded":"2024-04-23T18:13:06.663985+0000","last_fullsized":"2024-04-23T18:13:06.663985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315326+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:13:07.315326+0000","last_peered":"2024-04-23T18:13:07.315326+0000","last_clean":"2024-04-23T18:13:07.315326+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:13:07.315326+0000","last_undegraded":"2024-04-23T18:13:07.315326+0000","last_fullsized":"2024-04-23T18:13:07.315326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512846+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:13:06.512846+0000","last_peered":"2024-04-23T18:13:06.512846+0000","last_clean":"2024-04-23T18:13:06.512846+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:13:06.512846+0000","last_undegraded":"2024-04-23T18:13:06.512846+0000","last_fullsized":"2024-04-23T18:13:06.512846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512863+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:13:06.512863+0000","last_peered":"2024-04-23T18:13:06.512863+0000","last_clean":"2024-04-23T18:13:06.512863+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:13:06.512863+0000","last_undegraded":"2024-04-23T18:13:06.512863+0000","last_fullsized":"2024-04-23T18:13:06.512863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315259+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:13:07.315259+0000","last_peered":"2024-04-23T18:13:07.315259+0000","last_clean":"2024-04-23T18:13:07.315259+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:13:07.315259+0000","last_undegraded":"2024-04-23T18:13:07.315259+0000","last_fullsized":"2024-04-23T18:13:07.315259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256754+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:13:07.256754+0000","last_peered":"2024-04-23T18:13:07.256754+0000","last_clean":"2024-04-23T18:13:07.256754+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:13:07.256754+0000","last_undegraded":"2024-04-23T18:13:07.256754+0000","last_fullsized":"2024-04-23T18:13:07.256754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513015+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:13:06.513015+0000","last_peered":"2024-04-23T18:13:06.513015+0000","last_clean":"2024-04-23T18:13:06.513015+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:13:06.513015+0000","last_undegraded":"2024-04-23T18:13:06.513015+0000","last_fullsized":"2024-04-23T18:13:06.513015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255690+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:13:07.255690+0000","last_peered":"2024-04-23T18:13:07.255690+0000","last_clean":"2024-04-23T18:13:07.255690+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:13:07.255690+0000","last_undegraded":"2024-04-23T18:13:07.255690+0000","last_fullsized":"2024-04-23T18:13:07.255690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511541+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:13:06.511541+0000","last_peered":"2024-04-23T18:13:06.511541+0000","last_clean":"2024-04-23T18:13:06.511541+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:13:06.511541+0000","last_undegraded":"2024-04-23T18:13:06.511541+0000","last_fullsized":"2024-04-23T18:13:06.511541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512418+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:13:06.512418+0000","last_peered":"2024-04-23T18:13:06.512418+0000","last_clean":"2024-04-23T18:13:06.512418+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:13:06.512418+0000","last_undegraded":"2024-04-23T18:13:06.512418+0000","last_fullsized":"2024-04-23T18:13:06.512418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135150+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:13:07.135150+0000","last_peered":"2024-04-23T18:13:07.135150+0000","last_clean":"2024-04-23T18:13:07.135150+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:13:07.135150+0000","last_undegraded":"2024-04-23T18:13:07.135150+0000","last_fullsized":"2024-04-23T18:13:07.135150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039130+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:13:08.039130+0000","last_peered":"2024-04-23T18:13:08.039130+0000","last_clean":"2024-04-23T18:13:08.039130+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:13:08.039130+0000","last_undegraded":"2024-04-23T18:13:08.039130+0000","last_fullsized":"2024-04-23T18:13:08.039130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664465+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:13:06.664465+0000","last_peered":"2024-04-23T18:13:06.664465+0000","last_clean":"2024-04-23T18:13:06.664465+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:13:06.664465+0000","last_undegraded":"2024-04-23T18:13:06.664465+0000","last_fullsized":"2024-04-23T18:13:06.664465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039850+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:13:08.039850+0000","last_peered":"2024-04-23T18:13:08.039850+0000","last_clean":"2024-04-23T18:13:08.039850+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:13:08.039850+0000","last_undegraded":"2024-04-23T18:13:08.039850+0000","last_fullsized":"2024-04-23T18:13:08.039850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256401+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:13:07.256401+0000","last_peered":"2024-04-23T18:13:07.256401+0000","last_clean":"2024-04-23T18:13:07.256401+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:13:07.256401+0000","last_undegraded":"2024-04-23T18:13:07.256401+0000","last_fullsized":"2024-04-23T18:13:07.256401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513008+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:13:06.513008+0000","last_peered":"2024-04-23T18:13:06.513008+0000","last_clean":"2024-04-23T18:13:06.513008+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:13:06.513008+0000","last_undegraded":"2024-04-23T18:13:06.513008+0000","last_fullsized":"2024-04-23T18:13:06.513008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664283+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:06.664283+0000","last_peered":"2024-04-23T18:13:06.664283+0000","last_clean":"2024-04-23T18:13:06.664283+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:13:06.664283+0000","last_undegraded":"2024-04-23T18:13:06.664283+0000","last_fullsized":"2024-04-23T18:13:06.664283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.038123+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:13:08.038123+0000","last_peered":"2024-04-23T18:13:08.038123+0000","last_clean":"2024-04-23T18:13:08.038123+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:13:08.038123+0000","last_undegraded":"2024-04-23T18:13:08.038123+0000","last_fullsized":"2024-04-23T18:13:08.038123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315311+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:13:07.315311+0000","last_peered":"2024-04-23T18:13:07.315311+0000","last_clean":"2024-04-23T18:13:07.315311+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:13:07.315311+0000","last_undegraded":"2024-04-23T18:13:07.315311+0000","last_fullsized":"2024-04-23T18:13:07.315311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255834+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:13:07.255834+0000","last_peered":"2024-04-23T18:13:07.255834+0000","last_clean":"2024-04-23T18:13:07.255834+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:13:07.255834+0000","last_undegraded":"2024-04-23T18:13:07.255834+0000","last_fullsized":"2024-04-23T18:13:07.255834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255679+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:13:07.255679+0000","last_peered":"2024-04-23T18:13:07.255679+0000","last_clean":"2024-04-23T18:13:07.255679+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:13:07.255679+0000","last_undegraded":"2024-04-23T18:13:07.255679+0000","last_fullsized":"2024-04-23T18:13:07.255679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512635+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:13:06.512635+0000","last_peered":"2024-04-23T18:13:06.512635+0000","last_clean":"2024-04-23T18:13:06.512635+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:13:06.512635+0000","last_undegraded":"2024-04-23T18:13:06.512635+0000","last_fullsized":"2024-04-23T18:13:06.512635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256616+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:07.256616+0000","last_peered":"2024-04-23T18:13:07.256616+0000","last_clean":"2024-04-23T18:13:07.256616+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:13:07.256616+0000","last_undegraded":"2024-04-23T18:13:07.256616+0000","last_fullsized":"2024-04-23T18:13:07.256616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513322+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:13:06.513322+0000","last_peered":"2024-04-23T18:13:06.513322+0000","last_clean":"2024-04-23T18:13:06.513322+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:13:06.513322+0000","last_undegraded":"2024-04-23T18:13:06.513322+0000","last_fullsized":"2024-04-23T18:13:06.513322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256999+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:13:07.256999+0000","last_peered":"2024-04-23T18:13:07.256999+0000","last_clean":"2024-04-23T18:13:07.256999+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:13:07.256999+0000","last_undegraded":"2024-04-23T18:13:07.256999+0000","last_fullsized":"2024-04-23T18:13:07.256999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.038467+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:13:08.038467+0000","last_peered":"2024-04-23T18:13:08.038467+0000","last_clean":"2024-04-23T18:13:08.038467+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:13:08.038467+0000","last_undegraded":"2024-04-23T18:13:08.038467+0000","last_fullsized":"2024-04-23T18:13:08.038467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513039+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:06.513039+0000","last_peered":"2024-04-23T18:13:06.513039+0000","last_clean":"2024-04-23T18:13:06.513039+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:13:06.513039+0000","last_undegraded":"2024-04-23T18:13:06.513039+0000","last_fullsized":"2024-04-23T18:13:06.513039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315463+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:13:07.315463+0000","last_peered":"2024-04-23T18:13:07.315463+0000","last_clean":"2024-04-23T18:13:07.315463+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:13:07.315463+0000","last_undegraded":"2024-04-23T18:13:07.315463+0000","last_fullsized":"2024-04-23T18:13:07.315463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256403+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:07.256403+0000","last_peered":"2024-04-23T18:13:07.256403+0000","last_clean":"2024-04-23T18:13:07.256403+0000","last_became_active":"2024-04-23T18:13:05.534622+0000","last_became_peered":"2024-04-23T18:13:05.534622+0000","last_unstale":"2024-04-23T18:13:07.256403+0000","last_undegraded":"2024-04-23T18:13:07.256403+0000","last_fullsized":"2024-04-23T18:13:07.256403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:45:59.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.135310+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:13:07.135310+0000","last_peered":"2024-04-23T18:13:07.135310+0000","last_clean":"2024-04-23T18:13:07.135310+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:13:07.135310+0000","last_undegraded":"2024-04-23T18:13:07.135310+0000","last_fullsized":"2024-04-23T18:13:07.135310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256675+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:13:07.256675+0000","last_peered":"2024-04-23T18:13:07.256675+0000","last_clean":"2024-04-23T18:13:07.256675+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:13:07.256675+0000","last_undegraded":"2024-04-23T18:13:07.256675+0000","last_fullsized":"2024-04-23T18:13:07.256675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257087+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:13:07.257087+0000","last_peered":"2024-04-23T18:13:07.257087+0000","last_clean":"2024-04-23T18:13:07.257087+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:13:07.257087+0000","last_undegraded":"2024-04-23T18:13:07.257087+0000","last_fullsized":"2024-04-23T18:13:07.257087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257291+0000","last_change":"2024-04-23T18:13:05.532689+0000","last_active":"2024-04-23T18:13:07.257291+0000","last_peered":"2024-04-23T18:13:07.257291+0000","last_clean":"2024-04-23T18:13:07.257291+0000","last_became_active":"2024-04-23T18:13:05.532383+0000","last_became_peered":"2024-04-23T18:13:05.532383+0000","last_unstale":"2024-04-23T18:13:07.257291+0000","last_undegraded":"2024-04-23T18:13:07.257291+0000","last_fullsized":"2024-04-23T18:13:07.257291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:42.869965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512650+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:13:06.512650+0000","last_peered":"2024-04-23T18:13:06.512650+0000","last_clean":"2024-04-23T18:13:06.512650+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:13:06.512650+0000","last_undegraded":"2024-04-23T18:13:06.512650+0000","last_fullsized":"2024-04-23T18:13:06.512650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255860+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:13:07.255860+0000","last_peered":"2024-04-23T18:13:07.255860+0000","last_clean":"2024-04-23T18:13:07.255860+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:13:07.255860+0000","last_undegraded":"2024-04-23T18:13:07.255860+0000","last_fullsized":"2024-04-23T18:13:07.255860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511964+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:13:06.511964+0000","last_peered":"2024-04-23T18:13:06.511964+0000","last_clean":"2024-04-23T18:13:06.511964+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:13:06.511964+0000","last_undegraded":"2024-04-23T18:13:06.511964+0000","last_fullsized":"2024-04-23T18:13:06.511964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513580+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:13:06.513580+0000","last_peered":"2024-04-23T18:13:06.513580+0000","last_clean":"2024-04-23T18:13:06.513580+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:13:06.513580+0000","last_undegraded":"2024-04-23T18:13:06.513580+0000","last_fullsized":"2024-04-23T18:13:06.513580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":351,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256355+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:13:07.256355+0000","last_peered":"2024-04-23T18:13:07.256355+0000","last_clean":"2024-04-23T18:13:07.256355+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:13:07.256355+0000","last_undegraded":"2024-04-23T18:13:07.256355+0000","last_fullsized":"2024-04-23T18:13:07.256355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255536+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:13:07.255536+0000","last_peered":"2024-04-23T18:13:07.255536+0000","last_clean":"2024-04-23T18:13:07.255536+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:13:07.255536+0000","last_undegraded":"2024-04-23T18:13:07.255536+0000","last_fullsized":"2024-04-23T18:13:07.255536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.047857+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:13:09.047857+0000","last_peered":"2024-04-23T18:13:09.047857+0000","last_clean":"2024-04-23T18:13:09.047857+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:13:09.047857+0000","last_undegraded":"2024-04-23T18:13:09.047857+0000","last_fullsized":"2024-04-23T18:13:09.047857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.037646+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:13:08.037646+0000","last_peered":"2024-04-23T18:13:08.037646+0000","last_clean":"2024-04-23T18:13:08.037646+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:13:08.037646+0000","last_undegraded":"2024-04-23T18:13:08.037646+0000","last_fullsized":"2024-04-23T18:13:08.037646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255836+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:13:07.255836+0000","last_peered":"2024-04-23T18:13:07.255836+0000","last_clean":"2024-04-23T18:13:07.255836+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:13:07.255836+0000","last_undegraded":"2024-04-23T18:13:07.255836+0000","last_fullsized":"2024-04-23T18:13:07.255836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:09.049776+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:13:09.049776+0000","last_peered":"2024-04-23T18:13:09.049776+0000","last_clean":"2024-04-23T18:13:09.049776+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:13:09.049776+0000","last_undegraded":"2024-04-23T18:13:09.049776+0000","last_fullsized":"2024-04-23T18:13:09.049776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511816+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:13:06.511816+0000","last_peered":"2024-04-23T18:13:06.511816+0000","last_clean":"2024-04-23T18:13:06.511816+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:13:06.511816+0000","last_undegraded":"2024-04-23T18:13:06.511816+0000","last_fullsized":"2024-04-23T18:13:06.511816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.513650+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:13:06.513650+0000","last_peered":"2024-04-23T18:13:06.513650+0000","last_clean":"2024-04-23T18:13:06.513650+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:13:06.513650+0000","last_undegraded":"2024-04-23T18:13:06.513650+0000","last_fullsized":"2024-04-23T18:13:06.513650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256388+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:13:07.256388+0000","last_peered":"2024-04-23T18:13:07.256388+0000","last_clean":"2024-04-23T18:13:07.256388+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:13:07.256388+0000","last_undegraded":"2024-04-23T18:13:07.256388+0000","last_fullsized":"2024-04-23T18:13:07.256388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.663869+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:13:06.663869+0000","last_peered":"2024-04-23T18:13:06.663869+0000","last_clean":"2024-04-23T18:13:06.663869+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:13:06.663869+0000","last_undegraded":"2024-04-23T18:13:06.663869+0000","last_fullsized":"2024-04-23T18:13:06.663869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.315053+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:13:07.315053+0000","last_peered":"2024-04-23T18:13:07.315053+0000","last_clean":"2024-04-23T18:13:07.315053+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:13:07.315053+0000","last_undegraded":"2024-04-23T18:13:07.315053+0000","last_fullsized":"2024-04-23T18:13:07.315053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.040686+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:13:08.040686+0000","last_peered":"2024-04-23T18:13:08.040686+0000","last_clean":"2024-04-23T18:13:08.040686+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:13:08.040686+0000","last_undegraded":"2024-04-23T18:13:08.040686+0000","last_fullsized":"2024-04-23T18:13:08.040686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255864+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:13:07.255864+0000","last_peered":"2024-04-23T18:13:07.255864+0000","last_clean":"2024-04-23T18:13:07.255864+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:13:07.255864+0000","last_undegraded":"2024-04-23T18:13:07.255864+0000","last_fullsized":"2024-04-23T18:13:07.255864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256399+0000","last_change":"2024-04-23T18:13:05.023543+0000","last_active":"2024-04-23T18:13:07.256399+0000","last_peered":"2024-04-23T18:13:07.256399+0000","last_clean":"2024-04-23T18:13:07.256399+0000","last_became_active":"2024-04-23T18:13:05.023066+0000","last_became_peered":"2024-04-23T18:13:05.023066+0000","last_unstale":"2024-04-23T18:13:07.256399+0000","last_undegraded":"2024-04-23T18:13:07.256399+0000","last_fullsized":"2024-04-23T18:13:07.256399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:09.234277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512015+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:13:06.512015+0000","last_peered":"2024-04-23T18:13:06.512015+0000","last_clean":"2024-04-23T18:13:06.512015+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:13:06.512015+0000","last_undegraded":"2024-04-23T18:13:06.512015+0000","last_fullsized":"2024-04-23T18:13:06.512015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:08.039831+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:13:08.039831+0000","last_peered":"2024-04-23T18:13:08.039831+0000","last_clean":"2024-04-23T18:13:08.039831+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:13:08.039831+0000","last_undegraded":"2024-04-23T18:13:08.039831+0000","last_fullsized":"2024-04-23T18:13:08.039831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257245+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:13:07.257245+0000","last_peered":"2024-04-23T18:13:07.257245+0000","last_clean":"2024-04-23T18:13:07.257245+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:13:07.257245+0000","last_undegraded":"2024-04-23T18:13:07.257245+0000","last_fullsized":"2024-04-23T18:13:07.257245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256026+0000","last_change":"2024-04-23T18:13:05.020176+0000","last_active":"2024-04-23T18:13:07.256026+0000","last_peered":"2024-04-23T18:13:07.256026+0000","last_clean":"2024-04-23T18:13:07.256026+0000","last_became_active":"2024-04-23T18:13:05.019646+0000","last_became_peered":"2024-04-23T18:13:05.019646+0000","last_unstale":"2024-04-23T18:13:07.256026+0000","last_undegraded":"2024-04-23T18:13:07.256026+0000","last_fullsized":"2024-04-23T18:13:07.256026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.296751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256971+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:13:07.256971+0000","last_peered":"2024-04-23T18:13:07.256971+0000","last_clean":"2024-04-23T18:13:07.256971+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:13:07.256971+0000","last_undegraded":"2024-04-23T18:13:07.256971+0000","last_fullsized":"2024-04-23T18:13:07.256971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512663+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:13:06.512663+0000","last_peered":"2024-04-23T18:13:06.512663+0000","last_clean":"2024-04-23T18:13:06.512663+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:13:06.512663+0000","last_undegraded":"2024-04-23T18:13:06.512663+0000","last_fullsized":"2024-04-23T18:13:06.512663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.257096+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:13:07.257096+0000","last_peered":"2024-04-23T18:13:07.257096+0000","last_clean":"2024-04-23T18:13:07.257096+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:13:07.257096+0000","last_undegraded":"2024-04-23T18:13:07.257096+0000","last_fullsized":"2024-04-23T18:13:07.257096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255968+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:13:07.255968+0000","last_peered":"2024-04-23T18:13:07.255968+0000","last_clean":"2024-04-23T18:13:07.255968+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:13:07.255968+0000","last_undegraded":"2024-04-23T18:13:07.255968+0000","last_fullsized":"2024-04-23T18:13:07.255968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256897+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:13:07.256897+0000","last_peered":"2024-04-23T18:13:07.256897+0000","last_clean":"2024-04-23T18:13:07.256897+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:13:07.256897+0000","last_undegraded":"2024-04-23T18:13:07.256897+0000","last_fullsized":"2024-04-23T18:13:07.256897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255948+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:13:07.255948+0000","last_peered":"2024-04-23T18:13:07.255948+0000","last_clean":"2024-04-23T18:13:07.255948+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:13:07.255948+0000","last_undegraded":"2024-04-23T18:13:07.255948+0000","last_fullsized":"2024-04-23T18:13:07.255948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255841+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:13:07.255841+0000","last_peered":"2024-04-23T18:13:07.255841+0000","last_clean":"2024-04-23T18:13:07.255841+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:13:07.255841+0000","last_undegraded":"2024-04-23T18:13:07.255841+0000","last_fullsized":"2024-04-23T18:13:07.255841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256466+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:13:07.256466+0000","last_peered":"2024-04-23T18:13:07.256466+0000","last_clean":"2024-04-23T18:13:07.256466+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:13:07.256466+0000","last_undegraded":"2024-04-23T18:13:07.256466+0000","last_fullsized":"2024-04-23T18:13:07.256466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256006+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:13:07.256006+0000","last_peered":"2024-04-23T18:13:07.256006+0000","last_clean":"2024-04-23T18:13:07.256006+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:13:07.256006+0000","last_undegraded":"2024-04-23T18:13:07.256006+0000","last_fullsized":"2024-04-23T18:13:07.256006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.255671+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:13:07.255671+0000","last_peered":"2024-04-23T18:13:07.255671+0000","last_clean":"2024-04-23T18:13:07.255671+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:13:07.255671+0000","last_undegraded":"2024-04-23T18:13:07.255671+0000","last_fullsized":"2024-04-23T18:13:07.255671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256223+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:13:07.256223+0000","last_peered":"2024-04-23T18:13:07.256223+0000","last_clean":"2024-04-23T18:13:07.256223+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:13:07.256223+0000","last_undegraded":"2024-04-23T18:13:07.256223+0000","last_fullsized":"2024-04-23T18:13:07.256223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256148+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:13:07.256148+0000","last_peered":"2024-04-23T18:13:07.256148+0000","last_clean":"2024-04-23T18:13:07.256148+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:13:07.256148+0000","last_undegraded":"2024-04-23T18:13:07.256148+0000","last_fullsized":"2024-04-23T18:13:07.256148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511804+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:13:06.511804+0000","last_peered":"2024-04-23T18:13:06.511804+0000","last_clean":"2024-04-23T18:13:06.511804+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:13:06.511804+0000","last_undegraded":"2024-04-23T18:13:06.511804+0000","last_fullsized":"2024-04-23T18:13:06.511804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.512527+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:13:06.512527+0000","last_peered":"2024-04-23T18:13:06.512527+0000","last_clean":"2024-04-23T18:13:06.512527+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:13:06.512527+0000","last_undegraded":"2024-04-23T18:13:06.512527+0000","last_fullsized":"2024-04-23T18:13:06.512527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.511570+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:13:06.511570+0000","last_peered":"2024-04-23T18:13:06.511570+0000","last_clean":"2024-04-23T18:13:06.511570+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:13:06.511570+0000","last_undegraded":"2024-04-23T18:13:06.511570+0000","last_fullsized":"2024-04-23T18:13:06.511570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:06.664082+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:13:06.664082+0000","last_peered":"2024-04-23T18:13:06.664082+0000","last_clean":"2024-04-23T18:13:06.664082+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:13:06.664082+0000","last_undegraded":"2024-04-23T18:13:06.664082+0000","last_fullsized":"2024-04-23T18:13:06.664082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256078+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:13:07.256078+0000","last_peered":"2024-04-23T18:13:07.256078+0000","last_clean":"2024-04-23T18:13:07.256078+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:13:07.256078+0000","last_undegraded":"2024-04-23T18:13:07.256078+0000","last_fullsized":"2024-04-23T18:13:07.256078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:13:07.256133+0000","last_change":"2024-04-23T18:13:05.536877+0000","last_active":"2024-04-23T18:13:07.256133+0000","last_peered":"2024-04-23T18:13:07.256133+0000","last_clean":"2024-04-23T18:13:07.256133+0000","last_became_active":"2024-04-23T18:13:05.534611+0000","last_became_peered":"2024-04-23T18:13:05.534611+0000","last_unstale":"2024-04-23T18:13:07.256133+0000","last_undegraded":"2024-04-23T18:13:07.256133+0000","last_fullsized":"2024-04-23T18:13:07.256133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:55.986092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249130,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2612,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":443597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":1035951,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":443833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":937984,"data_stored":443641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1034587,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":925696,"data_stored":443015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36268,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708884,"statfs":{"total":95995035648,"available":95957897216,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":929792,"data_stored":436091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":443299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":434467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1033565,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:13:13.258 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T18:13:13.259 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T18:13:13.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T18:13:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:13 smithi040 ceph-mon[32626]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s rd, 2.5 KiB/s wr, 11 op/s 2024-04-23T18:13:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:13 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3832488145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:13 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3189915733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:13 smithi040 ceph-mon[32626]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:13 smithi064 ceph-mon[33062]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s rd, 2.5 KiB/s wr, 11 op/s 2024-04-23T18:13:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:13 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3832488145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:13 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3189915733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:13 smithi064 ceph-mon[33062]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:14 smithi032 ceph-mon[29221]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:13:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:14 smithi032 ceph-mon[29221]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:14.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:14 smithi040 ceph-mon[32626]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:13:14.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:14 smithi040 ceph-mon[32626]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:14 smithi064 ceph-mon[33062]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:13:14.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:14 smithi064 ceph-mon[33062]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:13:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:15 smithi032 ceph-mon[29221]: pgmap v295: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.6 KiB/s wr, 21 op/s 2024-04-23T18:13:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 -- 172.21.15.32:0/360643169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 msgr2=0x7fb18c10af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 --2- 172.21.15.32:0/360643169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c10af10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0099c0 tx=0x7fb17c009cd0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 -- 172.21.15.32:0/360643169 shutdown_connections 2024-04-23T18:13:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 --2- 172.21.15.32:0/360643169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c10af10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 --2- 172.21.15.32:0/360643169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c102ac0 0x7fb18c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 --2- 172.21.15.32:0/360643169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c103cc0 0x7fb18c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 -- 172.21.15.32:0/360643169 >> 172.21.15.32:0/360643169 conn(0x7fb18c0fe060 msgr2=0x7fb18c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 -- 172.21.15.32:0/360643169 shutdown_connections 2024-04-23T18:13:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.296+0000 7fb190890700 1 -- 172.21.15.32:0/360643169 wait complete. 2024-04-23T18:13:15.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 Processor -- start 2024-04-23T18:13:15.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 -- start start 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c102ac0 0x7fb18c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c103cc0 0x7fb18c19b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10e560 con 0x7fb18c104660 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10e3e0 con 0x7fb18c103cc0 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb190890700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10e6e0 con 0x7fb18c102ac0 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43506/0 (socket says 172.21.15.32:43506) 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 -- 172.21.15.32:0/234580213 learned_addr learned my addr 172.21.15.32:0/234580213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb189d9b700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c103cc0 0x7fb18c19b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18a59c700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c102ac0 0x7fb18c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 -- 172.21.15.32:0/234580213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c102ac0 msgr2=0x7fb18c19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:15.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c102ac0 0x7fb18c19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 -- 172.21.15.32:0/234580213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c103cc0 msgr2=0x7fb18c19b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c103cc0 0x7fb18c19b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.297+0000 7fb18ad9d700 1 -- 172.21.15.32:0/234580213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c1a2930 con 0x7fb18c104660 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb18ad9d700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0059d0 tx=0x7fb17c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c027070 con 0x7fb18c104660 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c004160 con 0x7fb18c104660 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00f460 con 0x7fb18c104660 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c1a2bc0 con 0x7fb18c104660 2024-04-23T18:13:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.298+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a3070 con 0x7fb18c104660 2024-04-23T18:13:15.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.299+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c050f80 con 0x7fb18c104660 2024-04-23T18:13:15.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.300+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb17c02a070 con 0x7fb18c104660 2024-04-23T18:13:15.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.301+0000 7fb1837fe700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 0x7fb1780a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:15.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.301+0000 7fb18a59c700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 0x7fb1780a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:15.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.301+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb17c0d1350 con 0x7fb18c104660 2024-04-23T18:13:15.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.301+0000 7fb18a59c700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 0x7fb1780a62c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb18c19c1c0 tx=0x7fb184009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:15.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.306+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb17c05eff0 con 0x7fb18c104660 2024-04-23T18:13:15.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:15 smithi040 ceph-mon[32626]: pgmap v295: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.6 KiB/s wr, 21 op/s 2024-04-23T18:13:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:15 smithi064 ceph-mon[33062]: pgmap v295: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.6 KiB/s wr, 21 op/s 2024-04-23T18:13:15.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.603+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fb18c1a34f0 con 0x7fb18c104660 2024-04-23T18:13:15.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.605+0000 7fb1837fe700 1 -- 172.21.15.32:0/234580213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fb18c1a34f0 con 0x7fb18c104660 2024-04-23T18:13:15.599 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-04-23T18:13:15.599 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQA7+idmnm4IJBAAReFoIfeEL3z2HbCCmsKtpw== 2024-04-23T18:13:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 msgr2=0x7fb1780a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 0x7fb1780a62c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb18c19c1c0 tx=0x7fb184009040 comp rx=0 tx=0).stop 2024-04-23T18:13:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 msgr2=0x7fb18c1a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0059d0 tx=0x7fb17c0047b0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 shutdown_connections 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb18c104660 0x7fb18c1a2260 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c102ac0 0x7fb18c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb1780a3e00 0x7fb1780a62c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 --2- 172.21.15.32:0/234580213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb18c103cc0 0x7fb18c19b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 >> 172.21.15.32:0/234580213 conn(0x7fb18c0fe060 msgr2=0x7fb18c074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 shutdown_connections 2024-04-23T18:13:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:15.607+0000 7fb190890700 1 -- 172.21.15.32:0/234580213 wait complete. 2024-04-23T18:13:16.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:13:16.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:13:16.290 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:13:16.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T18:13:16.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T18:13:16.329 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T18:13:16.329 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T18:13:16.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T18:13:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:13:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:13:16.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:13:16.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/234580213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:13:16.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:17.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:17 smithi032 ceph-mon[29221]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-23T18:13:17.320 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:17 smithi040 ceph-mon[32626]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-23T18:13:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:17 smithi064 ceph-mon[33062]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 -- 172.21.15.32:0/4279659800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc1026e0 msgr2=0x7f68dc10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 --2- 172.21.15.32:0/4279659800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc1026e0 0x7f68dc10e770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f68d400b0a0 tx=0x7f68d400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 -- 172.21.15.32:0/4279659800 shutdown_connections 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 --2- 172.21.15.32:0/4279659800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc1026e0 0x7f68dc10e770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 --2- 172.21.15.32:0/4279659800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc106990 0x7f68dc106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 --2- 172.21.15.32:0/4279659800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc101d20 0x7f68dc1021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 -- 172.21.15.32:0/4279659800 >> 172.21.15.32:0/4279659800 conn(0x7f68dc0788f0 msgr2=0x7f68dc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 -- 172.21.15.32:0/4279659800 shutdown_connections 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.339+0000 7f68e20e1700 1 -- 172.21.15.32:0/4279659800 wait complete. 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 Processor -- start 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 -- start start 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc101d20 0x7f68dc199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc1026e0 0x7f68dc19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc111b50 con 0x7f68dc106990 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1119d0 con 0x7f68dc101d20 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68e20e1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc111cd0 con 0x7f68dc1026e0 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68dbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.340+0000 7f68dbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48064/0 (socket says 172.21.15.32:48064) 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 -- 172.21.15.32:0/3811556292 learned_addr learned my addr 172.21.15.32:0/3811556292 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68daffd700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc1026e0 0x7f68dc19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68db7fe700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc101d20 0x7f68dc199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 -- 172.21.15.32:0/3811556292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc1026e0 msgr2=0x7f68dc19a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc1026e0 0x7f68dc19a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 -- 172.21.15.32:0/3811556292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc101d20 msgr2=0x7f68dc199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc101d20 0x7f68dc199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 -- 172.21.15.32:0/3811556292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc1a5550 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68dbfff700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f68d400b830 tx=0x7f68d4004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4021070 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68d4003c00 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68dc1a5720 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4004730 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.341+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68dc1a5b20 con 0x7f68dc106990 2024-04-23T18:13:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.342+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68dc103080 con 0x7f68dc106990 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.343+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f68d4007a00 con 0x7f68dc106990 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.344+0000 7f68d8ff9700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 0x7f68c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.344+0000 7f68db7fe700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 0x7f68c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.344+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f68d400f020 con 0x7f68dc106990 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.345+0000 7f68db7fe700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 0x7f68c80a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68cc006fd0 tx=0x7f68cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.349+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68d402a020 con 0x7f68dc106990 2024-04-23T18:13:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.617+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f68dc02df80 con 0x7f68dc106990 2024-04-23T18:13:18.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.621+0000 7f68d8ff9700 1 -- 172.21.15.32:0/3811556292 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264996 (secure 0 0 0) 0x7f68d409e590 con 0x7f68dc106990 2024-04-23T18:13:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 msgr2=0x7f68c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 0x7f68c80a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68cc006fd0 tx=0x7f68cc008040 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 msgr2=0x7f68dc1a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f68d400b830 tx=0x7f68d4004d90 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 shutdown_connections 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68dc106990 0x7f68dc1a5010 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f68c80a3cd0 0x7f68c80a6190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68dc101d20 0x7f68dc199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 --2- 172.21.15.32:0/3811556292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68dc1026e0 0x7f68dc19a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.624+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 >> 172.21.15.32:0/3811556292 conn(0x7f68dc0788f0 msgr2=0x7f68dc0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.625+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 shutdown_connections 2024-04-23T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:18.625+0000 7f68e20e1700 1 -- 172.21.15.32:0/3811556292 wait complete. 2024-04-23T18:13:19.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:19 smithi032 ceph-mon[29221]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:13:19.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3811556292' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:19 smithi040 ceph-mon[32626]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:13:19.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:19 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3811556292' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:19 smithi064 ceph-mon[33062]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:13:19.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:19 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3811556292' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3652066649},{"type":"v1","addr":"172.21.15.32:6801","nonce":3652066649}]},"active_addr":"172.21.15.32:6801/3652066649","active_change":"2024-04-23T18:05:55.984485+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":4149207751}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3162050466}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":993405612}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1504393466}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":941067918}]}]} 2024-04-23T18:13:19.464 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T18:13:19.464 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T18:13:19.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:13:19.715 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 -- 172.21.15.32:0/2995489705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 msgr2=0x7f3988108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2995489705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f3988108e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3978009990 tx=0x7f3978009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 -- 172.21.15.32:0/2995489705 shutdown_connections 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2995489705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 0x7f398810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2995489705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f3988108e10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2995489705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 0x7f398806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 -- 172.21.15.32:0/2995489705 >> 172.21.15.32:0/2995489705 conn(0x7f39880788f0 msgr2=0x7f3988078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 -- 172.21.15.32:0/2995489705 shutdown_connections 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.364+0000 7f398eb3d700 1 -- 172.21.15.32:0/2995489705 wait complete. 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 Processor -- start 2024-04-23T18:13:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 -- start start 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 0x7f39881990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 0x7f3988199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3988112bd0 con 0x7f3988108a30 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3988112a50 con 0x7f398806a750 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398eb3d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3988112d50 con 0x7f398806b0f0 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.365+0000 7f398d0da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48078/0 (socket says 172.21.15.32:48078) 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f3987fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 0x7f3988199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 -- 172.21.15.32:0/2983268778 learned_addr learned my addr 172.21.15.32:0/2983268778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398c8d9700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 0x7f39881990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 -- 172.21.15.32:0/2983268778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 msgr2=0x7f3988199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:21.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 0x7f3988199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 -- 172.21.15.32:0/2983268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 msgr2=0x7f39881990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 0x7f39881990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 -- 172.21.15.32:0/2983268778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398819e0a0 con 0x7f3988108a30 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398d0da700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3980007ee0 tx=0x7f398000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398000dbd0 con 0x7f3988108a30 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f398000dd70 con 0x7f3988108a30 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398819e390 con 0x7f3988108a30 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980005990 con 0x7f3988108a30 2024-04-23T18:13:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.366+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39881a5c10 con 0x7f3988108a30 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.367+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3988069730 con 0x7f3988108a30 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.368+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f398001a020 con 0x7f3988108a30 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.369+0000 7f3985ffb700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 0x7f39700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.369+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f39800cfb60 con 0x7f3988108a30 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.369+0000 7f398c8d9700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 0x7f39700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.369+0000 7f398c8d9700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 0x7f39700a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3978006010 tx=0x7f397801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.373+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3980010070 con 0x7f3988108a30 2024-04-23T18:13:21.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:21 smithi032 ceph-mon[29221]: pgmap v298: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-23T18:13:21.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:21 smithi064 ceph-mon[33062]: pgmap v298: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-23T18:13:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.586+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f398802df80 con 0x7f3988108a30 2024-04-23T18:13:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.587+0000 7f3985ffb700 1 -- 172.21.15.32:0/2983268778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22148 (secure 0 0 0) 0x7f398009c590 con 0x7f3988108a30 2024-04-23T18:13:21.582 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:21.582 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:06.508628+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 msgr2=0x7f39700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 0x7f39700a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3978006010 tx=0x7f397801a040 comp rx=0 tx=0).stop 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 msgr2=0x7f398819d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3980007ee0 tx=0x7f398000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 shutdown_connections 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3988108a30 0x7f398819d960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f39700a3cd0 0x7f39700a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398806a750 0x7f39881990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 --2- 172.21.15.32:0/2983268778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f398806b0f0 0x7f3988199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 >> 172.21.15.32:0/2983268778 conn(0x7f39880788f0 msgr2=0x7f39880ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 shutdown_connections 2024-04-23T18:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:21.590+0000 7f398eb3d700 1 -- 172.21.15.32:0/2983268778 wait complete. 2024-04-23T18:13:21.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:21 smithi040 ceph-mon[32626]: pgmap v298: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-23T18:13:22.363 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T18:13:22.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:13:22.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:22 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2983268778' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:22.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:22 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2983268778' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:22.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:22 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2983268778' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:23.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:23 smithi032 ceph-mon[29221]: pgmap v299: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-04-23T18:13:23.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:23 smithi064 ceph-mon[33062]: pgmap v299: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-04-23T18:13:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:23 smithi040 ceph-mon[32626]: pgmap v299: 97 pgs: 97 active+clean; 628 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-04-23T18:13:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 -- 172.21.15.32:0/3666199684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464107920 msgr2=0x7f4464107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 --2- 172.21.15.32:0/3666199684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464107920 0x7f4464107d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4454009990 tx=0x7f4454009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 -- 172.21.15.32:0/3666199684 shutdown_connections 2024-04-23T18:13:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 --2- 172.21.15.32:0/3666199684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464102330 0x7f446410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 --2- 172.21.15.32:0/3666199684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464107920 0x7f4464107d00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 --2- 172.21.15.32:0/3666199684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464101970 0x7f4464101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 -- 172.21.15.32:0/3666199684 >> 172.21.15.32:0/3666199684 conn(0x7f44640788f0 msgr2=0x7f4464078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 -- 172.21.15.32:0/3666199684 shutdown_connections 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 -- 172.21.15.32:0/3666199684 wait complete. 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.355+0000 7f446a0c2700 1 Processor -- start 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 -- start start 2024-04-23T18:13:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464101970 0x7f4464197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464107920 0x7f4464198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4464111b30 con 0x7f4464102330 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44641119b0 con 0x7f4464101970 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f446a0c2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4464111cb0 con 0x7f4464107920 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48088/0 (socket says 172.21.15.32:48088) 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 -- 172.21.15.32:0/415179546 learned_addr learned my addr 172.21.15.32:0/415179546 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4463fff700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464107920 0x7f4464198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f44637fe700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464101970 0x7f4464197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 -- 172.21.15.32:0/415179546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464107920 msgr2=0x7f4464198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464107920 0x7f4464198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 -- 172.21.15.32:0/415179546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464101970 msgr2=0x7f4464197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464101970 0x7f4464197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.356+0000 7f4462ffd700 1 -- 172.21.15.32:0/415179546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44641a75f0 con 0x7f4464102330 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f4462ffd700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9c0 tx=0x7f445800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445800cbd0 con 0x7f4464102330 2024-04-23T18:13:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44641a78e0 con 0x7f4464102330 2024-04-23T18:13:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f445800cd70 con 0x7f4464102330 2024-04-23T18:13:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4458005680 con 0x7f4464102330 2024-04-23T18:13:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.357+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44641a7dd0 con 0x7f4464102330 2024-04-23T18:13:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.358+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4464109eb0 con 0x7f4464102330 2024-04-23T18:13:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.359+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4458004750 con 0x7f4464102330 2024-04-23T18:13:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.360+0000 7f4460ff9700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 0x7f444c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.360+0000 7f44637fe700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 0x7f444c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.360+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f44580d0840 con 0x7f4464102330 2024-04-23T18:13:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.360+0000 7f44637fe700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 0x7f444c0a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4454006010 tx=0x7f445401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.363+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4458053ff0 con 0x7f4464102330 2024-04-23T18:13:24.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.575+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f446402df80 con 0x7f4464102330 2024-04-23T18:13:24.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.577+0000 7f4460ff9700 1 -- 172.21.15.32:0/415179546 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22148 (secure 0 0 0) 0x7f445809d270 con 0x7f4464102330 2024-04-23T18:13:24.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:13:24.571 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:06.508628+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:13:24.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 msgr2=0x7f444c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 0x7f444c0a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4454006010 tx=0x7f445401a040 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 msgr2=0x7f4464198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9c0 tx=0x7f445800ecd0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 shutdown_connections 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4464107920 0x7f4464198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f444c0a3cd0 0x7f444c0a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4464101970 0x7f4464197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 --2- 172.21.15.32:0/415179546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4464102330 0x7f4464198320 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 >> 172.21.15.32:0/415179546 conn(0x7f44640788f0 msgr2=0x7f44640ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 shutdown_connections 2024-04-23T18:13:24.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:24.579+0000 7f446a0c2700 1 -- 172.21.15.32:0/415179546 wait complete. 2024-04-23T18:13:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:25 smithi032 ceph-mon[29221]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-04-23T18:13:25.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:25 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/415179546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:25.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T18:13:25.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T18:13:25.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T18:13:25.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T18:13:25.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T18:13:25.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T18:13:25.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T18:13:25.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T18:13:25.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T18:13:25.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T18:13:25.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T18:13:25.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T18:13:25.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:25 smithi064 ceph-mon[33062]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-04-23T18:13:25.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:25 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/415179546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:25 smithi040 ceph-mon[32626]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-04-23T18:13:25.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:25 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/415179546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:13:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.320 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.344 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.345 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:26.352 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:27.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:27 smithi032 ceph-mon[29221]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-23T18:13:27.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:27 smithi064 ceph-mon[33062]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-23T18:13:27.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:27 smithi040 ceph-mon[32626]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-23T18:13:29.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:29 smithi032 ceph-mon[29221]: pgmap v302: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-23T18:13:29.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:29 smithi064 ceph-mon[33062]: pgmap v302: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-23T18:13:29.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:29 smithi040 ceph-mon[32626]: pgmap v302: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-23T18:13:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:31.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:31 smithi032 ceph-mon[29221]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-23T18:13:31.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:31 smithi064 ceph-mon[33062]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-23T18:13:31.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:31 smithi040 ceph-mon[32626]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-23T18:13:33.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:33 smithi032 ceph-mon[29221]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:13:33.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:33 smithi064 ceph-mon[33062]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:13:33.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:33 smithi040 ceph-mon[32626]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:13:35.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:35 smithi032 ceph-mon[29221]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-23T18:13:35.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:35 smithi064 ceph-mon[33062]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-23T18:13:35.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- 172.21.15.32:0/558001965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406b1f0 msgr2=0x7f182406b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 --2- 172.21.15.32:0/558001965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406b1f0 0x7f182406b670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1818009990 tx=0x7f1818009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- 172.21.15.32:0/558001965 shutdown_connections 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 --2- 172.21.15.32:0/558001965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182406bbb0 0x7f182410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 --2- 172.21.15.32:0/558001965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1824108a60 0x7f1824108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 --2- 172.21.15.32:0/558001965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406b1f0 0x7f182406b670 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- 172.21.15.32:0/558001965 >> 172.21.15.32:0/558001965 conn(0x7f18240789e0 msgr2=0x7f1824078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- 172.21.15.32:0/558001965 shutdown_connections 2024-04-23T18:13:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- 172.21.15.32:0/558001965 wait complete. 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 Processor -- start 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.503+0000 7f1829bdb700 1 -- start start 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182406b1f0 0x7f182419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 0x7f182419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1824108a60 0x7f182419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1824112ca0 con 0x7f182406bbb0 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1824112b20 con 0x7f182406b1f0 2024-04-23T18:13:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1829bdb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1824112e20 con 0x7f1824108a60 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 0x7f182419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 0x7f182419b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52364/0 (socket says 172.21.15.32:52364) 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 -- 172.21.15.32:0/606601192 learned_addr learned my addr 172.21.15.32:0/606601192 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f18237fe700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182406b1f0 0x7f182419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1823fff700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1824108a60 0x7f182419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 -- 172.21.15.32:0/606601192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1824108a60 msgr2=0x7f182419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1824108a60 0x7f182419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 -- 172.21.15.32:0/606601192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182406b1f0 msgr2=0x7f182419b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182406b1f0 0x7f182419b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18241a0290 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1822ffd700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 0x7f182419b7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f18180057c0 tx=0x7f18180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.504+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818027070 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.505+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1818003f20 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.505+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18241a0520 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.505+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818022b30 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.505+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18241a7e00 con 0x7f182406bbb0 2024-04-23T18:13:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.505+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18240631d0 con 0x7f182406bbb0 2024-04-23T18:13:35.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.507+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f181802a070 con 0x7f182406bbb0 2024-04-23T18:13:35.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.507+0000 7f1820ff9700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f180c0a3cd0 0x7f180c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:35.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.507+0000 7f18237fe700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f180c0a3cd0 0x7f180c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.507+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f18180d1af0 con 0x7f182406bbb0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f1820ff9700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] conn(0x7f180c0ad4b0 0x7f180c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1818009030 con 0x7f180c0ad4b0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f18180d1fd0 con 0x7f182406bbb0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f1823fff700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] conn(0x7f180c0ad4b0 0x7f180c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f18237fe700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f180c0a3cd0 0x7f180c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f181400abd0 tx=0x7f181400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.508+0000 7f1823fff700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] conn(0x7f180c0ad4b0 0x7f180c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:35.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.509+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== osd.0 v2:172.21.15.32:6802/1995005420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1818009030 con 0x7f180c0ad4b0 2024-04-23T18:13:35.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.525+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 --> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1824063650 con 0x7f180c0ad4b0 2024-04-23T18:13:35.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1820ff9700 1 -- 172.21.15.32:0/606601192 <== osd.0 v2:172.21.15.32:6802/1995005420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1824063650 con 0x7f180c0ad4b0 2024-04-23T18:13:35.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] conn(0x7f180c0ad4b0 msgr2=0x7f180c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] conn(0x7f180c0ad4b0 0x7f180c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f180c0a3cd0 msgr2=0x7f180c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f180c0a3cd0 0x7f180c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f181400abd0 tx=0x7f181400a380 comp rx=0 tx=0).stop 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 msgr2=0x7f182419b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 --2- 172.21.15.32:0/606601192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182406bbb0 0x7f182419b7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f18180057c0 tx=0x7f18180057f0 comp rx=0 tx=0).stop 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1823fff700 1 -- 172.21.15.32:0/606601192 reap_dead start 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 shutdown_connections 2024-04-23T18:13:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 >> 172.21.15.32:0/606601192 conn(0x7f18240789e0 msgr2=0x7f182410dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:35.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 shutdown_connections 2024-04-23T18:13:35.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:35.526+0000 7f1829bdb700 1 -- 172.21.15.32:0/606601192 wait complete. 2024-04-23T18:13:35.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:35 smithi040 ceph-mon[32626]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-23T18:13:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 -- 172.21.15.32:0/1225663620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec101940 msgr2=0x7f4aec101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 --2- 172.21.15.32:0/1225663620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec101940 0x7f4aec101dc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009990 tx=0x7f4ad8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 -- 172.21.15.32:0/1225663620 shutdown_connections 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 --2- 172.21.15.32:0/1225663620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aec102300 0x7f4aec10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 --2- 172.21.15.32:0/1225663620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4aec107990 0x7f4aec107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 --2- 172.21.15.32:0/1225663620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec101940 0x7f4aec101dc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 -- 172.21.15.32:0/1225663620 >> 172.21.15.32:0/1225663620 conn(0x7f4aec0788f0 msgr2=0x7f4aec078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 -- 172.21.15.32:0/1225663620 shutdown_connections 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.119+0000 7f4af315e700 1 -- 172.21.15.32:0/1225663620 wait complete. 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 Processor -- start 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 -- start start 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4aec101940 0x7f4aec199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 0x7f4aec19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aec107990 0x7f4aec1a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec111b00 con 0x7f4aec102300 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec111980 con 0x7f4aec107990 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af315e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec111c80 con 0x7f4aec101940 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 0x7f4aec19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 0x7f4aec19a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45758/0 (socket says 172.21.15.32:45758) 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4aebfff700 1 -- 172.21.15.32:0/2632846759 learned_addr learned my addr 172.21.15.32:0/2632846759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.120+0000 7f4af0efa700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4aec101940 0x7f4aec199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4af16fb700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aec107990 0x7f4aec1a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 -- 172.21.15.32:0/2632846759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4aec101940 msgr2=0x7f4aec199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4aec101940 0x7f4aec199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 -- 172.21.15.32:0/2632846759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aec107990 msgr2=0x7f4aec1a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aec107990 0x7f4aec1a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec1a5520 con 0x7f4aec102300 2024-04-23T18:13:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4aebfff700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 0x7f4aec19a4a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8000c00 tx=0x7f4ad80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8027070 con 0x7f4aec102300 2024-04-23T18:13:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ad8003f20 con 0x7f4aec102300 2024-04-23T18:13:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8022cf0 con 0x7f4aec102300 2024-04-23T18:13:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aec1a5750 con 0x7f4aec102300 2024-04-23T18:13:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.121+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aec1a5b50 con 0x7f4aec102300 2024-04-23T18:13:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.122+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4aec063110 con 0x7f4aec102300 2024-04-23T18:13:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4ad802a070 con 0x7f4aec102300 2024-04-23T18:13:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4ae9ffb700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4af0efa700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4ad80d1760 con 0x7f4aec102300 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4ae9ffb700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] conn(0x7f4adc0ad4b0 0x7f4adc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.124+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ad800ce90 con 0x7f4adc0ad4b0 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.125+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f4ad80d1c40 con 0x7f4aec102300 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.125+0000 7f4af16fb700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] conn(0x7f4adc0ad4b0 0x7f4adc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.125+0000 7f4af0efa700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00045b0 tx=0x7f4ae000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.125+0000 7f4af16fb700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] conn(0x7f4adc0ad4b0 0x7f4adc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.126+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== osd.9 v2:172.21.15.32:6826/3112793650 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ad800ce90 con 0x7f4adc0ad4b0 2024-04-23T18:13:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.142+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 --> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4aec0635c0 con 0x7f4adc0ad4b0 2024-04-23T18:13:37.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.142+0000 7f4ae9ffb700 1 -- 172.21.15.32:0/2632846759 <== osd.9 v2:172.21.15.32:6826/3112793650 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4aec0635c0 con 0x7f4adc0ad4b0 2024-04-23T18:13:37.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] conn(0x7f4adc0ad4b0 msgr2=0x7f4adc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] conn(0x7f4adc0ad4b0 0x7f4adc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4adc0a3cd0 msgr2=0x7f4adc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00045b0 tx=0x7f4ae000b3c0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 msgr2=0x7f4aec19a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 --2- 172.21.15.32:0/2632846759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aec102300 0x7f4aec19a4a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8000c00 tx=0x7f4ad80047b0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af16fb700 1 -- 172.21.15.32:0/2632846759 reap_dead start 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 shutdown_connections 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 >> 172.21.15.32:0/2632846759 conn(0x7f4aec0788f0 msgr2=0x7f4aec0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 shutdown_connections 2024-04-23T18:13:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:37.143+0000 7f4af315e700 1 -- 172.21.15.32:0/2632846759 wait complete. 2024-04-23T18:13:37.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:37 smithi032 ceph-mon[29221]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:13:37.418 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:37 smithi040 ceph-mon[32626]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:13:37.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:37 smithi064 ceph-mon[33062]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:13:39.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:39 smithi032 ceph-mon[29221]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:39.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:39 smithi064 ceph-mon[33062]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:39.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:39 smithi040 ceph-mon[32626]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:40.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 -- 172.21.15.32:0/2293355443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4101970 msgr2=0x7f5ce4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 --2- 172.21.15.32:0/2293355443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4101970 0x7f5ce4101df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009990 tx=0x7f5cd4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 -- 172.21.15.32:0/2293355443 shutdown_connections 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 --2- 172.21.15.32:0/2293355443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce4102330 0x7f5ce410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 --2- 172.21.15.32:0/2293355443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce4107920 0x7f5ce4107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 --2- 172.21.15.32:0/2293355443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4101970 0x7f5ce4101df0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 -- 172.21.15.32:0/2293355443 >> 172.21.15.32:0/2293355443 conn(0x7f5ce4078910 msgr2=0x7f5ce4078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 -- 172.21.15.32:0/2293355443 shutdown_connections 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.353+0000 7f5ceb778700 1 -- 172.21.15.32:0/2293355443 wait complete. 2024-04-23T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 Processor -- start 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 -- start start 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce4101970 0x7f5ce41034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce4102330 0x7f5ce4103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 0x7f5ce4103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4111b30 con 0x7f5ce4107920 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41119b0 con 0x7f5ce4102330 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ceb778700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4111cb0 con 0x7f5ce4101970 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ce9d15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 0x7f5ce4103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ce9d15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 0x7f5ce4103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45790/0 (socket says 172.21.15.32:45790) 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.354+0000 7f5ce9d15700 1 -- 172.21.15.32:0/223530747 learned_addr learned my addr 172.21.15.32:0/223530747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce8d13700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce4102330 0x7f5ce4103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9514700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce4101970 0x7f5ce41034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 -- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce4101970 msgr2=0x7f5ce41034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce4101970 0x7f5ce41034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 -- 172.21.15.32:0/223530747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce4102330 msgr2=0x7f5ce4103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce4102330 0x7f5ce4103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce41a7680 con 0x7f5ce4107920 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ce9d15700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 0x7f5ce4103f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00ada0 tx=0x7f5cdc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc017020 con 0x7f5ce4107920 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc00ce30 con 0x7f5ce4107920 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce41a7970 con 0x7f5ce4107920 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc0054a0 con 0x7f5ce4107920 2024-04-23T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.355+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41a7eb0 con 0x7f5ce4107920 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.356+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5ce40630d0 con 0x7f5ce4107920 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.357+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5cdc00c980 con 0x7f5ce4107920 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5cda7fc700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5cd00a3cd0 0x7f5cd00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5ce9514700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5cd00a3cd0 0x7f5cd00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5cdc0d0630 con 0x7f5ce4107920 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5cda7fc700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] conn(0x7f5cd00ad4b0 0x7f5cd00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cd00aff80 con 0x7f5cd00ad4b0 2024-04-23T18:13:40.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f5cdc0d09e0 con 0x7f5ce4107920 2024-04-23T18:13:40.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.358+0000 7f5ce9514700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5cd00a3cd0 0x7f5cd00a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0005950 tx=0x7f5ce00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:40.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.359+0000 7f5ce8d13700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] conn(0x7f5cd00ad4b0 0x7f5cd00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:40.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.359+0000 7f5ce8d13700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] conn(0x7f5cd00ad4b0 0x7f5cd00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:40.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.360+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== osd.2 v2:172.21.15.64:6800/3950848968 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cd00aff80 con 0x7f5cd00ad4b0 2024-04-23T18:13:40.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.376+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 --> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ce4063550 con 0x7f5cd00ad4b0 2024-04-23T18:13:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5cda7fc700 1 -- 172.21.15.32:0/223530747 <== osd.2 v2:172.21.15.64:6800/3950848968 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5ce4063550 con 0x7f5cd00ad4b0 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] conn(0x7f5cd00ad4b0 msgr2=0x7f5cd00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] conn(0x7f5cd00ad4b0 0x7f5cd00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5cd00a3cd0 msgr2=0x7f5cd00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f5cd00a3cd0 0x7f5cd00a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0005950 tx=0x7f5ce00058e0 comp rx=0 tx=0).stop 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 msgr2=0x7f5ce4103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 --2- 172.21.15.32:0/223530747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce4107920 0x7f5ce4103f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00ada0 tx=0x7f5cdc00c330 comp rx=0 tx=0).stop 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ce9d15700 1 -- 172.21.15.32:0/223530747 reap_dead start 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 shutdown_connections 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 >> 172.21.15.32:0/223530747 conn(0x7f5ce4078910 msgr2=0x7f5ce406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 shutdown_connections 2024-04-23T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:40.377+0000 7f5ceb778700 1 -- 172.21.15.32:0/223530747 wait complete. 2024-04-23T18:13:41.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:41 smithi064 ceph-mon[33062]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:41.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:41 smithi032 ceph-mon[29221]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:41.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:41 smithi040 ceph-mon[32626]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:13:41.714 INFO:teuthology.orchestra.run.smithi032.stdout:292057776159 2024-04-23T18:13:41.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T18:13:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 -- 172.21.15.32:0/1249219151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4103450 msgr2=0x7fd7b410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 --2- 172.21.15.32:0/1249219151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4103450 0x7fd7b410f8b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00b0a0 tx=0x7fd7ac00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 -- 172.21.15.32:0/1249219151 shutdown_connections 2024-04-23T18:13:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 --2- 172.21.15.32:0/1249219151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4103450 0x7fd7b410f8b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 --2- 172.21.15.32:0/1249219151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7b4108a50 0x7fd7b4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 --2- 172.21.15.32:0/1249219151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4102a90 0x7fd7b4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 -- 172.21.15.32:0/1249219151 >> 172.21.15.32:0/1249219151 conn(0x7fd7b4079ba0 msgr2=0x7fd7b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 -- 172.21.15.32:0/1249219151 shutdown_connections 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.200+0000 7fd7b9737700 1 -- 172.21.15.32:0/1249219151 wait complete. 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 Processor -- start 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 -- start start 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 0x7fd7b4199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4103450 0x7fd7b4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7b4108a50 0x7fd7b419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4112c00 con 0x7fd7b4102a90 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4112a80 con 0x7fd7b4103450 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b9737700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4112d80 con 0x7fd7b4108a50 2024-04-23T18:13:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 0x7fd7b4199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 0x7fd7b4199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45804/0 (socket says 172.21.15.32:45804) 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 -- 172.21.15.32:0/3311213956 learned_addr learned my addr 172.21.15.32:0/3311213956 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 -- 172.21.15.32:0/3311213956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7b4108a50 msgr2=0x7fd7b419d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7b4108a50 0x7fd7b419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b27fc700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4103450 0x7fd7b4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 -- 172.21.15.32:0/3311213956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4103450 msgr2=0x7fd7b4199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4103450 0x7fd7b4199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b2ffd700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b419dff0 con 0x7fd7b4102a90 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.201+0000 7fd7b27fc700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7b4103450 0x7fd7b4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7b2ffd700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 0x7fd7b4199030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd7a400d980 tx=0x7fd7a400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4017070 con 0x7fd7b4102a90 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a4004830 con 0x7fd7b4102a90 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b419e2e0 con 0x7fd7b4102a90 2024-04-23T18:13:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4005440 con 0x7fd7b4102a90 2024-04-23T18:13:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.202+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b41a5b60 con 0x7fd7b4102a90 2024-04-23T18:13:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.203+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7b40631d0 con 0x7fd7b4102a90 2024-04-23T18:13:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.204+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd7a40055e0 con 0x7fd7b4102a90 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7abfff700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd7a00a3cd0 0x7fd7a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7b27fc700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd7a00a3cd0 0x7fd7a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd7a40d0850 con 0x7fd7b4102a90 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7abfff700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] conn(0x7fd7a00ad4b0 0x7fd7a00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7a00aff80 con 0x7fd7a00ad4b0 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fd7a40d0d30 con 0x7fd7b4102a90 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7b37fe700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] conn(0x7fd7a00ad4b0 0x7fd7a00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.205+0000 7fd7b27fc700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd7a00a3cd0 0x7fd7a00a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005fd0 tx=0x7fd79c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.206+0000 7fd7b37fe700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] conn(0x7fd7a00ad4b0 0x7fd7a00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.206+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== osd.6 v2:172.21.15.32:6818/3191318575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7a00aff80 con 0x7fd7a00ad4b0 2024-04-23T18:13:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.223+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 --> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7b4063650 con 0x7fd7a00ad4b0 2024-04-23T18:13:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.223+0000 7fd7abfff700 1 -- 172.21.15.32:0/3311213956 <== osd.6 v2:172.21.15.32:6818/3191318575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd7b4063650 con 0x7fd7a00ad4b0 2024-04-23T18:13:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] conn(0x7fd7a00ad4b0 msgr2=0x7fd7a00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] conn(0x7fd7a00ad4b0 0x7fd7a00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd7a00a3cd0 msgr2=0x7fd7a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd7a00a3cd0 0x7fd7a00a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005fd0 tx=0x7fd79c005e20 comp rx=0 tx=0).stop 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 msgr2=0x7fd7b4199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 --2- 172.21.15.32:0/3311213956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b4102a90 0x7fd7b4199030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd7a400d980 tx=0x7fd7a400dd40 comp rx=0 tx=0).stop 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b37fe700 1 -- 172.21.15.32:0/3311213956 reap_dead start 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 shutdown_connections 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 >> 172.21.15.32:0/3311213956 conn(0x7fd7b4079ba0 msgr2=0x7fd7b41016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 shutdown_connections 2024-04-23T18:13:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.224+0000 7fd7b9737700 1 -- 172.21.15.32:0/3311213956 wait complete. 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 -- 172.21.15.32:0/725853457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0107990 msgr2=0x7feba0107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 --2- 172.21.15.32:0/725853457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0107990 0x7feba0107d70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feb90009990 tx=0x7feb90009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 -- 172.21.15.32:0/725853457 shutdown_connections 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 --2- 172.21.15.32:0/725853457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feba0102300 0x7feba010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 --2- 172.21.15.32:0/725853457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0107990 0x7feba0107d70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 --2- 172.21.15.32:0/725853457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0101940 0x7feba0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 -- 172.21.15.32:0/725853457 >> 172.21.15.32:0/725853457 conn(0x7feba00788f0 msgr2=0x7feba0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 -- 172.21.15.32:0/725853457 shutdown_connections 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.429+0000 7feba5889700 1 -- 172.21.15.32:0/725853457 wait complete. 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 Processor -- start 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 -- start start 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 0x7feba0103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feba0102300 0x7feba0103e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0107990 0x7feba01043c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feba0111b30 con 0x7feba0101940 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feba01119b0 con 0x7feba0107990 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feba5889700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feba0111cb0 con 0x7feba0102300 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 0x7feba0103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 0x7feba0103940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45818/0 (socket says 172.21.15.32:45818) 2024-04-23T18:13:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 -- 172.21.15.32:0/3718696614 learned_addr learned my addr 172.21.15.32:0/3718696614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9f7fe700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0107990 0x7feba01043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb967fc700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feba0102300 0x7feba0103e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 -- 172.21.15.32:0/3718696614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feba0102300 msgr2=0x7feba0103e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feba0102300 0x7feba0103e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 -- 172.21.15.32:0/3718696614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0107990 msgr2=0x7feba01043c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0107990 0x7feba01043c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9effd700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba01a7690 con 0x7feba0101940 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.430+0000 7feb9f7fe700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba0107990 0x7feba01043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feb9effd700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 0x7feba0103940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feb90005910 tx=0x7feb9000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb90027070 con 0x7feba0101940 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb90022470 con 0x7feba0101940 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba01a7920 con 0x7feba0101940 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb90017650 con 0x7feba0101940 2024-04-23T18:13:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.431+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba01a7dd0 con 0x7feba0101940 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.433+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feba006a970 con 0x7feba0101940 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.434+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feb9002a070 con 0x7feba0101940 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feba4887700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7feb8c0a3d50 0x7feb8c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feb967fc700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7feb8c0a3d50 0x7feb8c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7feb900d12c0 con 0x7feba0101940 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feba4887700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] conn(0x7feb8c0ad530 0x7feb8c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb8c0b0000 con 0x7feb8c0ad530 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7feb900d8050 con 0x7feba0101940 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.435+0000 7feb9f7fe700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] conn(0x7feb8c0ad530 0x7feb8c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.436+0000 7feb967fc700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7feb8c0a3d50 0x7feb8c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feba006b800 tx=0x7feb88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.436+0000 7feb9f7fe700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] conn(0x7feb8c0ad530 0x7feb8c0af950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.437+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== osd.3 v2:172.21.15.32:6810/2339361307 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb8c0b0000 con 0x7feb8c0ad530 2024-04-23T18:13:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.453+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 --> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feba0103060 con 0x7feb8c0ad530 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba4887700 1 -- 172.21.15.32:0/3718696614 <== osd.3 v2:172.21.15.32:6810/2339361307 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feba0103060 con 0x7feb8c0ad530 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] conn(0x7feb8c0ad530 msgr2=0x7feb8c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] conn(0x7feb8c0ad530 0x7feb8c0af950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7feb8c0a3d50 msgr2=0x7feb8c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7feb8c0a3d50 0x7feb8c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feba006b800 tx=0x7feb88009260 comp rx=0 tx=0).stop 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 msgr2=0x7feba0103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 --2- 172.21.15.32:0/3718696614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba0101940 0x7feba0103940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feb90005910 tx=0x7feb9000b670 comp rx=0 tx=0).stop 2024-04-23T18:13:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feb9f7fe700 1 -- 172.21.15.32:0/3718696614 reap_dead start 2024-04-23T18:13:42.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 shutdown_connections 2024-04-23T18:13:42.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 >> 172.21.15.32:0/3718696614 conn(0x7feba00788f0 msgr2=0x7feba0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:42.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 shutdown_connections 2024-04-23T18:13:42.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.454+0000 7feba5889700 1 -- 172.21.15.32:0/3718696614 wait complete. 2024-04-23T18:13:43.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:43 smithi032 ceph-mon[29221]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:43.377 INFO:teuthology.orchestra.run.smithi032.stdout:107374182462 2024-04-23T18:13:43.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T18:13:43.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:43 smithi064 ceph-mon[33062]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 -- 172.21.15.32:0/3153363679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 msgr2=0x7f8308108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3153363679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f8308108eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009990 tx=0x7f82f8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 -- 172.21.15.32:0/3153363679 shutdown_connections 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3153363679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 0x7f830810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3153363679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f8308108eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3153363679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8308101030 0x7f8308101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 -- 172.21.15.32:0/3153363679 >> 172.21.15.32:0/3153363679 conn(0x7f8308077fb0 msgr2=0x7f83080ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 -- 172.21.15.32:0/3153363679 shutdown_connections 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.931+0000 7f830e8cb700 1 -- 172.21.15.32:0/3153363679 wait complete. 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.932+0000 7f830e8cb700 1 Processor -- start 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.932+0000 7f830e8cb700 1 -- start start 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8308101030 0x7f8308199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 0x7f8308199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f830819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8308112c30 con 0x7f8308108ad0 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8308112ab0 con 0x7f8308101030 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830e8cb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8308112db0 con 0x7f83081019d0 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f830819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f830819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45838/0 (socket says 172.21.15.32:45838) 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 -- 172.21.15.32:0/3535808312 learned_addr learned my addr 172.21.15.32:0/3535808312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f8307fff700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8308101030 0x7f8308199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f83077fe700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 0x7f8308199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 -- 172.21.15.32:0/3535808312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 msgr2=0x7f8308199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 0x7f8308199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 -- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8308101030 msgr2=0x7f8308199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8308101030 0x7f8308199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830819e110 con 0x7f8308108ad0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f83077fe700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83081019d0 0x7f8308199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.933+0000 7f830ce68700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f830819d9d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8300007ee0 tx=0x7f830000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.934+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830000dbd0 con 0x7f8308108ad0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.934+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830000dd70 con 0x7f8308108ad0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.934+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830819e400 con 0x7f8308108ad0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.934+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300005990 con 0x7f8308108ad0 2024-04-23T18:13:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.934+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081a5c80 con 0x7f8308108ad0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.935+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83080631d0 con 0x7f8308108ad0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.936+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f830001a020 con 0x7f8308108ad0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.936+0000 7f83057fa700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f82f00a3cd0 0x7f82f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f8307fff700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f82f00a3cd0 0x7f82f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f83000cfb60 con 0x7f8308108ad0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f83057fa700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] conn(0x7f82f00ad4b0 0x7f82f00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f830000fec0 con 0x7f82f00ad4b0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f83000d22b0 con 0x7f8308108ad0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f8307fff700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f82f00a3cd0 0x7f82f00a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b5c0 tx=0x7f82f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.937+0000 7f83077fe700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] conn(0x7f82f00ad4b0 0x7f82f00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.938+0000 7f83077fe700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] conn(0x7f82f00ad4b0 0x7f82f00af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.938+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== osd.7 v2:172.21.15.40:6816/142928766 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f830000fec0 con 0x7f82f00ad4b0 2024-04-23T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.971+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 --> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8308063650 con 0x7f82f00ad4b0 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f83057fa700 1 -- 172.21.15.32:0/3535808312 <== osd.7 v2:172.21.15.40:6816/142928766 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8308063650 con 0x7f82f00ad4b0 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] conn(0x7f82f00ad4b0 msgr2=0x7f82f00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] conn(0x7f82f00ad4b0 0x7f82f00af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f82f00a3cd0 msgr2=0x7f82f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f82f00a3cd0 0x7f82f00a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b5c0 tx=0x7f82f8005fd0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 msgr2=0x7f830819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830e8cb700 1 --2- 172.21.15.32:0/3535808312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8308108ad0 0x7f830819d9d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8300007ee0 tx=0x7f830000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.972+0000 7f830ce68700 1 -- 172.21.15.32:0/3535808312 reap_dead start 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.973+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 shutdown_connections 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.973+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 >> 172.21.15.32:0/3535808312 conn(0x7f8308077fb0 msgr2=0x7f83080ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.973+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 shutdown_connections 2024-04-23T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:42.973+0000 7f830e8cb700 1 -- 172.21.15.32:0/3535808312 wait complete. 2024-04-23T18:13:43.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:43 smithi040 ceph-mon[32626]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:44.397 INFO:teuthology.orchestra.run.smithi032.stdout:34359738444 2024-04-23T18:13:44.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T18:13:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 -- 172.21.15.32:0/2942970835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 msgr2=0x7fb02c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 --2- 172.21.15.32:0/2942970835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 0x7fb02c10dfc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb02400b0d0 tx=0x7fb02400b3e0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 -- 172.21.15.32:0/2942970835 shutdown_connections 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 --2- 172.21.15.32:0/2942970835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 0x7fb02c10dfc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 --2- 172.21.15.32:0/2942970835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 0x7fb02c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 --2- 172.21.15.32:0/2942970835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 -- 172.21.15.32:0/2942970835 >> 172.21.15.32:0/2942970835 conn(0x7fb02c077fb0 msgr2=0x7fb02c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 -- 172.21.15.32:0/2942970835 shutdown_connections 2024-04-23T18:13:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.597+0000 7fb030824700 1 -- 172.21.15.32:0/2942970835 wait complete. 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 Processor -- start 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 -- start start 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 0x7fb02c0fd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c0fd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 0x7fb02c0fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c111b00 con 0x7fb02c1063f0 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c111980 con 0x7fb02c101360 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.598+0000 7fb030824700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c111c80 con 0x7fb02c101d10 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c0fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c0fd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39942/0 (socket says 172.21.15.32:39942) 2024-04-23T18:13:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 -- 172.21.15.32:0/1458361367 learned_addr learned my addr 172.21.15.32:0/1458361367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb02ad9d700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 0x7fb02c0fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb02a59c700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 0x7fb02c0fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 -- 172.21.15.32:0/1458361367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 msgr2=0x7fb02c0fd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 0x7fb02c0fd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 -- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 msgr2=0x7fb02c0fdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb02c1063f0 0x7fb02c0fdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c1a7610 con 0x7fb02c101d10 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb02a59c700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb02c101360 0x7fb02c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb029d9b700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c0fd960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb02000d980 tx=0x7fb02000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.599+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020017070 con 0x7fb02c101d10 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.600+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02c1a7900 con 0x7fb02c101d10 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.600+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb020009de0 con 0x7fb02c101d10 2024-04-23T18:13:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.600+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02000f3c0 con 0x7fb02c101d10 2024-04-23T18:13:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.600+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02c1a7e40 con 0x7fb02c101d10 2024-04-23T18:13:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.601+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb00c001030 con 0x7fb02c101d10 2024-04-23T18:13:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.602+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb020009930 con 0x7fb02c101d10 2024-04-23T18:13:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb01b7fe700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb0140a3cd0 0x7fb0140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb02a59c700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb0140a3cd0 0x7fb0140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb0200d0380 con 0x7fb02c101d10 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb01b7fe700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] conn(0x7fb0140ad4b0 0x7fb0140af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0140aff80 con 0x7fb0140ad4b0 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb0200d0730 con 0x7fb02c101d10 2024-04-23T18:13:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb02ad9d700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] conn(0x7fb0140ad4b0 0x7fb0140af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb02a59c700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb0140a3cd0 0x7fb0140a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb01c005950 tx=0x7fb01c005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.603+0000 7fb02ad9d700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] conn(0x7fb0140ad4b0 0x7fb0140af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.604+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== osd.5 v2:172.21.15.64:6808/1980421875 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0140aff80 con 0x7fb0140ad4b0 2024-04-23T18:13:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 --> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb00c002e60 con 0x7fb0140ad4b0 2024-04-23T18:13:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb01b7fe700 1 -- 172.21.15.32:0/1458361367 <== osd.5 v2:172.21.15.64:6808/1980421875 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb00c002e60 con 0x7fb0140ad4b0 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] conn(0x7fb0140ad4b0 msgr2=0x7fb0140af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] conn(0x7fb0140ad4b0 0x7fb0140af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb0140a3cd0 msgr2=0x7fb0140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fb0140a3cd0 0x7fb0140a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb01c005950 tx=0x7fb01c005f50 comp rx=0 tx=0).stop 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 msgr2=0x7fb02c0fd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.621+0000 7fb030824700 1 --2- 172.21.15.32:0/1458361367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb02c101d10 0x7fb02c0fd960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb02000d980 tx=0x7fb02000dc90 comp rx=0 tx=0).stop 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.622+0000 7fb02ad9d700 1 -- 172.21.15.32:0/1458361367 reap_dead start 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.622+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 shutdown_connections 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.622+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 >> 172.21.15.32:0/1458361367 conn(0x7fb02c077fb0 msgr2=0x7fb02c10eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.622+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 shutdown_connections 2024-04-23T18:13:44.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:44.622+0000 7fb030824700 1 -- 172.21.15.32:0/1458361367 wait complete. 2024-04-23T18:13:45.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:45 smithi032 ceph-mon[29221]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:45.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:45 smithi064 ceph-mon[33062]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:45.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:45 smithi040 ceph-mon[32626]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 -- 172.21.15.32:0/1496404885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec102ac0 msgr2=0x7f29ec102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 --2- 172.21.15.32:0/1496404885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec102ac0 0x7f29ec102f40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009990 tx=0x7f29e0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 -- 172.21.15.32:0/1496404885 shutdown_connections 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 --2- 172.21.15.32:0/1496404885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ec103480 0x7f29ec10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 --2- 172.21.15.32:0/1496404885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 0x7f29ec108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 --2- 172.21.15.32:0/1496404885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec102ac0 0x7f29ec102f40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 -- 172.21.15.32:0/1496404885 >> 172.21.15.32:0/1496404885 conn(0x7f29ec079bf0 msgr2=0x7f29ec07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 -- 172.21.15.32:0/1496404885 shutdown_connections 2024-04-23T18:13:46.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.147+0000 7f29f286b700 1 -- 172.21.15.32:0/1496404885 wait complete. 2024-04-23T18:13:46.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 Processor -- start 2024-04-23T18:13:46.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 -- start start 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ec102ac0 0x7f29ec074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 0x7f29ec075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 0x7f29ec0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec112cf0 con 0x7f29ec103480 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec112b70 con 0x7f29ec108ab0 2024-04-23T18:13:46.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29f286b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec112e70 con 0x7f29ec102ac0 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29eb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 0x7f29ec075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.148+0000 7f29eb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 0x7f29ec075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45874/0 (socket says 172.21.15.32:45874) 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 -- 172.21.15.32:0/996859215 learned_addr learned my addr 172.21.15.32:0/996859215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29ebfff700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ec102ac0 0x7f29ec074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29f0e08700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 0x7f29ec0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 -- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ec102ac0 msgr2=0x7f29ec074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ec102ac0 0x7f29ec074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 -- 172.21.15.32:0/996859215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 msgr2=0x7f29ec0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 0x7f29ec0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec1a7790 con 0x7f29ec103480 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29f0e08700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29ec108ab0 0x7f29ec0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29eb7fe700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 0x7f29ec075280 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29e00047b0 tx=0x7f29e0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0026070 con 0x7f29ec103480 2024-04-23T18:13:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e0004c90 con 0x7f29ec103480 2024-04-23T18:13:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1a7a20 con 0x7f29ec103480 2024-04-23T18:13:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0003d90 con 0x7f29ec103480 2024-04-23T18:13:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.149+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1a7e70 con 0x7f29ec103480 2024-04-23T18:13:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.150+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29ec0630d0 con 0x7f29ec103480 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29e0029070 con 0x7f29ec103480 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29d40a3cd0 0x7f29d40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29ebfff700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29d40a3cd0 0x7f29d40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f29e00d1450 con 0x7f29ec103480 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] conn(0x7f29d40ad4b0 0x7f29d40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29e0019e50 con 0x7f29d40ad4b0 2024-04-23T18:13:46.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.152+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f29e00d1930 con 0x7f29ec103480 2024-04-23T18:13:46.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.153+0000 7f29f0e08700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] conn(0x7f29d40ad4b0 0x7f29d40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.153+0000 7f29ebfff700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29d40a3cd0 0x7f29d40a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f29dc005950 tx=0x7f29dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.153+0000 7f29f0e08700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] conn(0x7f29d40ad4b0 0x7f29d40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.154+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== osd.8 v2:172.21.15.64:6816/3639114663 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29e0019e50 con 0x7f29d40ad4b0 2024-04-23T18:13:46.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.171+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 --> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f29ec0634d0 con 0x7f29d40ad4b0 2024-04-23T18:13:46.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.171+0000 7f29e97fa700 1 -- 172.21.15.32:0/996859215 <== osd.8 v2:172.21.15.64:6816/3639114663 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f29ec0634d0 con 0x7f29d40ad4b0 2024-04-23T18:13:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] conn(0x7f29d40ad4b0 msgr2=0x7f29d40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] conn(0x7f29d40ad4b0 0x7f29d40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29d40a3cd0 msgr2=0x7f29d40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f29d40a3cd0 0x7f29d40a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f29dc005950 tx=0x7f29dc00b410 comp rx=0 tx=0).stop 2024-04-23T18:13:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 msgr2=0x7f29ec075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 --2- 172.21.15.32:0/996859215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec103480 0x7f29ec075280 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29e00047b0 tx=0x7f29e0003680 comp rx=0 tx=0).stop 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f0e08700 1 -- 172.21.15.32:0/996859215 reap_dead start 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 shutdown_connections 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 >> 172.21.15.32:0/996859215 conn(0x7f29ec079bf0 msgr2=0x7f29ec100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 shutdown_connections 2024-04-23T18:13:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.172+0000 7f29f286b700 1 -- 172.21.15.32:0/996859215 wait complete. 2024-04-23T18:13:46.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 -- 172.21.15.32:0/159065845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c102320 msgr2=0x7f232c10e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 --2- 172.21.15.32:0/159065845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c102320 0x7f232c10e830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f232400b0a0 tx=0x7f232400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 -- 172.21.15.32:0/159065845 shutdown_connections 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 --2- 172.21.15.32:0/159065845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c102320 0x7f232c10e830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 --2- 172.21.15.32:0/159065845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f232c107960 0x7f232c107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 --2- 172.21.15.32:0/159065845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232c101960 0x7f232c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.936+0000 7f2332e8f700 1 -- 172.21.15.32:0/159065845 >> 172.21.15.32:0/159065845 conn(0x7f232c0789e0 msgr2=0x7f232c078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.937+0000 7f2332e8f700 1 -- 172.21.15.32:0/159065845 shutdown_connections 2024-04-23T18:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.937+0000 7f2332e8f700 1 -- 172.21.15.32:0/159065845 wait complete. 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.938+0000 7f2332e8f700 1 Processor -- start 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.938+0000 7f2332e8f700 1 -- start start 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.938+0000 7f2332e8f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232c101960 0x7f232c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2332e8f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f232c102320 0x7f232c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2332e8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 0x7f232c1a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2332e8f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f232c111c50 con 0x7f232c107960 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2332e8f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f232c111ad0 con 0x7f232c101960 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2332e8f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f232c111dd0 con 0x7f232c102320 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 0x7f232c1a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 0x7f232c1a5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34766/0 (socket says 172.21.15.32:34766) 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 -- 172.21.15.32:0/2188882689 learned_addr learned my addr 172.21.15.32:0/2188882689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f2330c2b700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232c101960 0x7f232c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f232bfff700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f232c102320 0x7f232c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 -- 172.21.15.32:0/2188882689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f232c102320 msgr2=0x7f232c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f232c102320 0x7f232c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 -- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232c101960 msgr2=0x7f232c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232c101960 0x7f232c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.939+0000 7f233142c700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c1a56b0 con 0x7f232c107960 2024-04-23T18:13:46.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f233142c700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 0x7f232c1a5170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f232400b070 tx=0x7f232400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324021070 con 0x7f232c107960 2024-04-23T18:13:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2324007b80 con 0x7f232c107960 2024-04-23T18:13:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c1a5880 con 0x7f232c107960 2024-04-23T18:13:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23240052a0 con 0x7f232c107960 2024-04-23T18:13:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.940+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c1a5d30 con 0x7f232c107960 2024-04-23T18:13:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.941+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f232c063150 con 0x7f232c107960 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.943+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2324024070 con 0x7f232c107960 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.943+0000 7f2329ffb700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f231c0a3d50 0x7f231c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.943+0000 7f2330c2b700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f231c0a3d50 0x7f231c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.943+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f232400f020 con 0x7f232c107960 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f2329ffb700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] conn(0x7f231c0ad530 0x7f231c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f232401fe80 con 0x7f231c0ad530 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f23240d1ff0 con 0x7f232c107960 2024-04-23T18:13:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f2330c2b700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f231c0a3d50 0x7f231c0a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2320009a10 tx=0x7f2320009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f232bfff700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] conn(0x7f231c0ad530 0x7f231c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:46.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.944+0000 7f232bfff700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] conn(0x7f231c0ad530 0x7f231c0af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:46.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.945+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== osd.4 v2:172.21.15.40:6808/3289252322 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f232401fe80 con 0x7f231c0ad530 2024-04-23T18:13:46.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.962+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 --> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f232c0635d0 con 0x7f231c0ad530 2024-04-23T18:13:46.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2329ffb700 1 -- 172.21.15.32:0/2188882689 <== osd.4 v2:172.21.15.40:6808/3289252322 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f232c0635d0 con 0x7f231c0ad530 2024-04-23T18:13:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] conn(0x7f231c0ad530 msgr2=0x7f231c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] conn(0x7f231c0ad530 0x7f231c0af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f231c0a3d50 msgr2=0x7f231c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f231c0a3d50 0x7f231c0a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2320009a10 tx=0x7f2320009190 comp rx=0 tx=0).stop 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 msgr2=0x7f232c1a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 --2- 172.21.15.32:0/2188882689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f232c107960 0x7f232c1a5170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f232400b070 tx=0x7f232400ce50 comp rx=0 tx=0).stop 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f233142c700 1 -- 172.21.15.32:0/2188882689 reap_dead start 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 shutdown_connections 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 >> 172.21.15.32:0/2188882689 conn(0x7f232c0789e0 msgr2=0x7f232c104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 shutdown_connections 2024-04-23T18:13:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:46.963+0000 7f2332e8f700 1 -- 172.21.15.32:0/2188882689 wait complete. 2024-04-23T18:13:47.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:47 smithi032 ceph-mon[29221]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:13:47.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:47 smithi064 ceph-mon[33062]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:13:47.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:47 smithi040 ceph-mon[32626]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:13:47.698 INFO:teuthology.orchestra.run.smithi032.stdout:257698037797 2024-04-23T18:13:47.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T18:13:48.309 INFO:teuthology.orchestra.run.smithi032.stdout:141733920826 2024-04-23T18:13:48.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T18:13:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 -- 172.21.15.32:0/2753017421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 msgr2=0x7f766c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 --2- 172.21.15.32:0/2753017421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c101db0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7658009990 tx=0x7f7658009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 -- 172.21.15.32:0/2753017421 shutdown_connections 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 --2- 172.21.15.32:0/2753017421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c1022f0 0x7f766c10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 --2- 172.21.15.32:0/2753017421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c107980 0x7f766c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 --2- 172.21.15.32:0/2753017421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c101db0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 -- 172.21.15.32:0/2753017421 >> 172.21.15.32:0/2753017421 conn(0x7f766c0788b0 msgr2=0x7f766c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 -- 172.21.15.32:0/2753017421 shutdown_connections 2024-04-23T18:13:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.513+0000 7f7673064700 1 -- 172.21.15.32:0/2753017421 wait complete. 2024-04-23T18:13:48.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 Processor -- start 2024-04-23T18:13:48.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 -- start start 2024-04-23T18:13:48.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c1022f0 0x7f766c199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c107980 0x7f766c19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f766c111ad0 con 0x7f766c101930 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f766c111950 con 0x7f766c1022f0 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7673064700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f766c111c50 con 0x7f766c107980 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7670e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7670e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34778/0 (socket says 172.21.15.32:34778) 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7670e00700 1 -- 172.21.15.32:0/1915627983 learned_addr learned my addr 172.21.15.32:0/1915627983 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f766bfff700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c1022f0 0x7f766c199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7670e00700 1 -- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c107980 msgr2=0x7f766c19d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.514+0000 7f7671601700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c107980 0x7f766c19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7670e00700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c107980 0x7f766c19d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7670e00700 1 -- 172.21.15.32:0/1915627983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c1022f0 msgr2=0x7f766c199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7670e00700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c1022f0 0x7f766c199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7670e00700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c19e000 con 0x7f766c101930 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f766bfff700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f766c1022f0 0x7f766c199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7671601700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f766c107980 0x7f766c19d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7670e00700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c198ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f766000c9b0 tx=0x7f766000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660007b00 con 0x7f766c101930 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7660007ca0 con 0x7f766c101930 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660018730 con 0x7f766c101930 2024-04-23T18:13:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.515+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c19e2f0 con 0x7f766c101930 2024-04-23T18:13:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.516+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c1a5b70 con 0x7f766c101930 2024-04-23T18:13:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.516+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f766c0631d0 con 0x7f766c101930 2024-04-23T18:13:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.518+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f766000f450 con 0x7f766c101930 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7669ffb700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f765c0a3d50 0x7f765c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f766bfff700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f765c0a3d50 0x7f765c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f76600d0890 con 0x7f766c101930 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7669ffb700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] conn(0x7f765c0ad530 0x7f765c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f765c0b0000 con 0x7f765c0ad530 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f76600d0c40 con 0x7f766c101930 2024-04-23T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f766bfff700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f765c0a3d50 0x7f765c0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7658006010 tx=0x7f765800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:48.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.519+0000 7f7671601700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] conn(0x7f765c0ad530 0x7f765c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:48.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.520+0000 7f7671601700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] conn(0x7f765c0ad530 0x7f765c0af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:48.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.521+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== osd.11 v2:172.21.15.64:6824/4083016219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f765c0b0000 con 0x7f765c0ad530 2024-04-23T18:13:48.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.541+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 --> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f766c0635d0 con 0x7f765c0ad530 2024-04-23T18:13:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7669ffb700 1 -- 172.21.15.32:0/1915627983 <== osd.11 v2:172.21.15.64:6824/4083016219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f766c0635d0 con 0x7f765c0ad530 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] conn(0x7f765c0ad530 msgr2=0x7f765c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] conn(0x7f765c0ad530 0x7f765c0af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f765c0a3d50 msgr2=0x7f765c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f765c0a3d50 0x7f765c0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7658006010 tx=0x7f765800b540 comp rx=0 tx=0).stop 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 msgr2=0x7f766c198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7673064700 1 --2- 172.21.15.32:0/1915627983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f766c101930 0x7f766c198ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f766000c9b0 tx=0x7f766000cd70 comp rx=0 tx=0).stop 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.542+0000 7f7671601700 1 -- 172.21.15.32:0/1915627983 reap_dead start 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.543+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 shutdown_connections 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.543+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 >> 172.21.15.32:0/1915627983 conn(0x7f766c0788b0 msgr2=0x7f766c06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.543+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 shutdown_connections 2024-04-23T18:13:48.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:48.543+0000 7f7673064700 1 -- 172.21.15.32:0/1915627983 wait complete. 2024-04-23T18:13:49.415 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:49 smithi032 ceph-mon[29221]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:13:49.417 INFO:teuthology.orchestra.run.smithi032.stdout:167503724597 2024-04-23T18:13:49.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T18:13:49.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:49 smithi064 ceph-mon[33062]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:13:49.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:49 smithi040 ceph-mon[32626]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:13:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.635+0000 7f59b81d3700 1 -- 172.21.15.32:0/2581483251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0103450 msgr2=0x7f59b010f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.635+0000 7f59b81d3700 1 --2- 172.21.15.32:0/2581483251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0103450 0x7f59b010f8b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59a800b0a0 tx=0x7f59a800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 -- 172.21.15.32:0/2581483251 shutdown_connections 2024-04-23T18:13:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 --2- 172.21.15.32:0/2581483251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0103450 0x7f59b010f8b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 --2- 172.21.15.32:0/2581483251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59b0108a50 0x7f59b0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 --2- 172.21.15.32:0/2581483251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b0102a90 0x7f59b0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 -- 172.21.15.32:0/2581483251 >> 172.21.15.32:0/2581483251 conn(0x7f59b0079ba0 msgr2=0x7f59b0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 -- 172.21.15.32:0/2581483251 shutdown_connections 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 -- 172.21.15.32:0/2581483251 wait complete. 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 Processor -- start 2024-04-23T18:13:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.636+0000 7f59b81d3700 1 -- start start 2024-04-23T18:13:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b0102a90 0x7f59b0199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59b0103450 0x7f59b0199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 0x7f59b019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0112c00 con 0x7f59b0108a50 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0112a80 con 0x7f59b0102a90 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0112d80 con 0x7f59b0103450 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 0x7f59b019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 0x7f59b019d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34800/0 (socket says 172.21.15.32:34800) 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 -- 172.21.15.32:0/743288912 learned_addr learned my addr 172.21.15.32:0/743288912 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b576e700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59b0103450 0x7f59b0199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b5f6f700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b0102a90 0x7f59b0199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 -- 172.21.15.32:0/743288912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59b0103450 msgr2=0x7f59b0199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59b0103450 0x7f59b0199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 -- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b0102a90 msgr2=0x7f59b0199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b0102a90 0x7f59b0199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b019e040 con 0x7f59b0108a50 2024-04-23T18:13:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b6770700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 0x7f59b019d900 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f59a800b830 tx=0x7f59a8004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8021070 con 0x7f59b0108a50 2024-04-23T18:13:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59a8007e30 con 0x7f59b0108a50 2024-04-23T18:13:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.637+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b019e2d0 con 0x7f59b0108a50 2024-04-23T18:13:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.638+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8004700 con 0x7f59b0108a50 2024-04-23T18:13:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.638+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b01a5bb0 con 0x7f59b0108a50 2024-04-23T18:13:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.639+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59b00631d0 con 0x7f59b0108a50 2024-04-23T18:13:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.640+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f59a8003d70 con 0x7f59b0108a50 2024-04-23T18:13:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59a6ffd700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f599c0a3cd0 0x7f599c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59b5f6f700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f599c0a3cd0 0x7f599c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f59a800f020 con 0x7f59b0108a50 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59a6ffd700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] conn(0x7f599c0ad4b0 0x7f599c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f599c0aff80 con 0x7f599c0ad4b0 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f59a80d1f70 con 0x7f59b0108a50 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59b5f6f700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f599c0a3cd0 0x7f599c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0045b0 tx=0x7f59ac00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.641+0000 7f59b576e700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] conn(0x7f599c0ad4b0 0x7f599c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.642+0000 7f59b576e700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] conn(0x7f599c0ad4b0 0x7f599c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.643+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== osd.10 v2:172.21.15.40:6824/4006751902 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f599c0aff80 con 0x7f599c0ad4b0 2024-04-23T18:13:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.660+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 --> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59b0063650 con 0x7f599c0ad4b0 2024-04-23T18:13:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.661+0000 7f59a6ffd700 1 -- 172.21.15.32:0/743288912 <== osd.10 v2:172.21.15.40:6824/4006751902 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f59b0063650 con 0x7f599c0ad4b0 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.661+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] conn(0x7f599c0ad4b0 msgr2=0x7f599c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.661+0000 7f59b81d3700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] conn(0x7f599c0ad4b0 0x7f599c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.661+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f599c0a3cd0 msgr2=0x7f599c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.661+0000 7f59b81d3700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f599c0a3cd0 0x7f599c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0045b0 tx=0x7f59ac00b3c0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 msgr2=0x7f59b019d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 --2- 172.21.15.32:0/743288912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59b0108a50 0x7f59b019d900 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f59a800b830 tx=0x7f59a8004ce0 comp rx=0 tx=0).stop 2024-04-23T18:13:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b6770700 1 -- 172.21.15.32:0/743288912 reap_dead start 2024-04-23T18:13:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 shutdown_connections 2024-04-23T18:13:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 >> 172.21.15.32:0/743288912 conn(0x7f59b0079ba0 msgr2=0x7f59b0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 shutdown_connections 2024-04-23T18:13:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:49.662+0000 7f59b81d3700 1 -- 172.21.15.32:0/743288912 wait complete. 2024-04-23T18:13:50.181 INFO:teuthology.orchestra.run.smithi032.stdout:77309411395 2024-04-23T18:13:50.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T18:13:50.817 INFO:teuthology.orchestra.run.smithi032.stdout:201863462959 2024-04-23T18:13:50.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T18:13:51.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:51 smithi064 ceph-mon[33062]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-23T18:13:51.435 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:51 smithi032 ceph-mon[29221]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-23T18:13:51.463 INFO:teuthology.orchestra.run.smithi032.stdout:313532612635 2024-04-23T18:13:51.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T18:13:51.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:51 smithi040 ceph-mon[32626]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-23T18:13:51.889 INFO:teuthology.orchestra.run.smithi032.stdout:223338299433 2024-04-23T18:13:51.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T18:13:52.731 INFO:teuthology.orchestra.run.smithi032.stdout:347892350997 2024-04-23T18:13:52.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 -- 172.21.15.32:0/2556700552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 msgr2=0x7ff4f8102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 --2- 172.21.15.32:0/2556700552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f8102f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009990 tx=0x7ff4e8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 -- 172.21.15.32:0/2556700552 shutdown_connections 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 --2- 172.21.15.32:0/2556700552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f8103480 0x7ff4f810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 --2- 172.21.15.32:0/2556700552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f8108ab0 0x7ff4f8108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 --2- 172.21.15.32:0/2556700552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f8102f40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 -- 172.21.15.32:0/2556700552 >> 172.21.15.32:0/2556700552 conn(0x7ff4f8079bf0 msgr2=0x7ff4f807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 -- 172.21.15.32:0/2556700552 shutdown_connections 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.078+0000 7ff5001ae700 1 -- 172.21.15.32:0/2556700552 wait complete. 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 Processor -- start 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 -- start start 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f81990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f8103480 0x7ff4f8199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f8108ab0 0x7ff4f819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8112cb0 con 0x7ff4f8102ac0 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8112b30 con 0x7ff4f8108ab0 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff5001ae700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8112e30 con 0x7ff4f8103480 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f81990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f81990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34832/0 (socket says 172.21.15.32:34832) 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 -- 172.21.15.32:0/3901005779 learned_addr learned my addr 172.21.15.32:0/3901005779 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fe74b700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f8108ab0 0x7ff4f819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fd749700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f8103480 0x7ff4f8199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 -- 172.21.15.32:0/3901005779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f8103480 msgr2=0x7ff4f8199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f8103480 0x7ff4f8199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 -- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f8108ab0 msgr2=0x7ff4f819d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f8108ab0 0x7ff4f819d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.079+0000 7ff4fdf4a700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f819e0a0 con 0x7ff4f8102ac0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff4fdf4a700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f81990e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c9b0 tx=0x7ff4f400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4007b00 con 0x7ff4f8102ac0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4f4007ca0 con 0x7ff4f8102ac0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff5001ae700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f819e390 con 0x7ff4f8102ac0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4018660 con 0x7ff4f8102ac0 2024-04-23T18:13:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.080+0000 7ff5001ae700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f81a5c10 con 0x7ff4f8102ac0 2024-04-23T18:13:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.081+0000 7ff5001ae700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4f8063150 con 0x7ff4f8102ac0 2024-04-23T18:13:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.082+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff4f400f450 con 0x7ff4f8102ac0 2024-04-23T18:13:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.083+0000 7ff4eeffd700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4e40a3d50 0x7ff4e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.083+0000 7ff4fd749700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4e40a3d50 0x7ff4e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.083+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff4f40d0720 con 0x7ff4f8102ac0 2024-04-23T18:13:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.083+0000 7ff4eeffd700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] conn(0x7ff4e40ad530 0x7ff4e40af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.083+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4f4014ec0 con 0x7ff4e40ad530 2024-04-23T18:13:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.084+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff4f40d0c00 con 0x7ff4f8102ac0 2024-04-23T18:13:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.084+0000 7ff4fd749700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4e40a3d50 0x7ff4e40a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006010 tx=0x7ff4e801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.084+0000 7ff4fe74b700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] conn(0x7ff4e40ad530 0x7ff4e40af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.085+0000 7ff4fe74b700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] conn(0x7ff4e40ad530 0x7ff4e40af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.086+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== osd.1 v2:172.21.15.40:6800/2541914444 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4f4014ec0 con 0x7ff4e40ad530 2024-04-23T18:13:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.100 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.101 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.101 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.121+0000 7ff5001ae700 1 -- 172.21.15.32:0/3901005779 --> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff4f8063580 con 0x7ff4e40ad530 2024-04-23T18:13:53.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.128+0000 7ff4eeffd700 1 -- 172.21.15.32:0/3901005779 <== osd.1 v2:172.21.15.40:6800/2541914444 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff4f8063580 con 0x7ff4e40ad530 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.133+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] conn(0x7ff4e40ad530 msgr2=0x7ff4e40af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.133+0000 7ff4ecff9700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] conn(0x7ff4e40ad530 0x7ff4e40af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4e40a3d50 msgr2=0x7ff4e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7ff4e40a3d50 0x7ff4e40a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006010 tx=0x7ff4e801a040 comp rx=0 tx=0).stop 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 msgr2=0x7ff4f81990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 --2- 172.21.15.32:0/3901005779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4f8102ac0 0x7ff4f81990e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c9b0 tx=0x7ff4f400ccc0 comp rx=0 tx=0).stop 2024-04-23T18:13:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4fe74b700 1 -- 172.21.15.32:0/3901005779 reap_dead start 2024-04-23T18:13:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 shutdown_connections 2024-04-23T18:13:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.134+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 >> 172.21.15.32:0/3901005779 conn(0x7ff4f8079bf0 msgr2=0x7ff4f8100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.136+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 shutdown_connections 2024-04-23T18:13:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:13:53.136+0000 7ff4ecff9700 1 -- 172.21.15.32:0/3901005779 wait complete. 2024-04-23T18:13:53.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:53 smithi032 ceph-mon[29221]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:13:53.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:53 smithi064 ceph-mon[33062]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:13:53.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:53 smithi040 ceph-mon[32626]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:13:55.450 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:55 smithi032 ceph-mon[29221]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:55.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:55 smithi040 ceph-mon[32626]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:55.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:55 smithi064 ceph-mon[33062]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:13:56.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:13:56.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:56 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-23T18:13:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:13:56.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:56 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-23T18:13:56.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-23T18:13:56.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-23T18:13:56.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T18:13:56.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-23T18:13:56.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:13:56.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:13:56.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:56 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-23T18:13:57.020 INFO:teuthology.orchestra.run.smithi032.stdout:55834574923 2024-04-23T18:13:57.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-23T18:13:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:57 smithi032 ceph-mon[29221]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:13:57.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:57.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-23T18:13:57.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:57 smithi040 ceph-mon[32626]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-23T18:13:57.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:57 smithi064 ceph-mon[33062]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:13:58.480 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:58 smithi032 ceph-mon[29221]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:13:58.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:58 smithi040 ceph-mon[32626]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:13:58.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:58 smithi064 ceph-mon[33062]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:13:59.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:13:59 smithi032 ceph-mon[29221]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:13:59.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:13:59 smithi040 ceph-mon[32626]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:13:59.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:13:59 smithi064 ceph-mon[33062]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:13:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 -- 172.21.15.32:0/1784297253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc107990 msgr2=0x7f8fcc107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 --2- 172.21.15.32:0/1784297253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc107990 0x7f8fcc107d70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009990 tx=0x7f8fbc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 -- 172.21.15.32:0/1784297253 shutdown_connections 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 --2- 172.21.15.32:0/1784297253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 0x7f8fcc10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 --2- 172.21.15.32:0/1784297253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc107990 0x7f8fcc107d70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 --2- 172.21.15.32:0/1784297253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc101940 0x7f8fcc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 -- 172.21.15.32:0/1784297253 >> 172.21.15.32:0/1784297253 conn(0x7f8fcc0788f0 msgr2=0x7f8fcc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 -- 172.21.15.32:0/1784297253 shutdown_connections 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.259+0000 7f8fd195d700 1 -- 172.21.15.32:0/1784297253 wait complete. 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 Processor -- start 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 -- start start 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:00.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 0x7f8fcc1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc107990 0x7f8fcc19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc111ae0 con 0x7f8fcc101940 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc111960 con 0x7f8fcc102300 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fd195d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc111c60 con 0x7f8fcc107990 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:00.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38148/0 (socket says 172.21.15.32:38148) 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.260+0000 7f8fcaffd700 1 -- 172.21.15.32:0/735497606 learned_addr learned my addr 172.21.15.32:0/735497606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcb7fe700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc107990 0x7f8fcc19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fca7fc700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 0x7f8fcc1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 -- 172.21.15.32:0/735497606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc107990 msgr2=0x7f8fcc19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc107990 0x7f8fcc19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 -- 172.21.15.32:0/735497606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 msgr2=0x7f8fcc1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 0x7f8fcc1995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 -- 172.21.15.32:0/735497606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc19e0c0 con 0x7f8fcc101940 2024-04-23T18:14:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fcaffd700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000c00 tx=0x7f8fbc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc027070 con 0x7f8fcc101940 2024-04-23T18:14:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fbc003fa0 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fcc19e350 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc022db0 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.261+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fcc1a5c30 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.262+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fcc103060 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.263+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8fbc02a070 con 0x7f8fcc101940 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.264+0000 7f8fd095b700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 0x7f8fb40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.264+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8fbc0d1650 con 0x7f8fcc101940 2024-04-23T18:14:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.264+0000 7f8fca7fc700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 0x7f8fb40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.264+0000 7f8fca7fc700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 0x7f8fb40a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005950 tx=0x7f8fc00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.267+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fbc013070 con 0x7f8fcc101940 2024-04-23T18:14:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.481+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f8fcc069730 con 0x7f8fcc101940 2024-04-23T18:14:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.481+0000 7f8fd095b700 1 -- 172.21.15.32:0/735497606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8fbc09df90 con 0x7f8fcc101940 2024-04-23T18:14:00.476 INFO:teuthology.orchestra.run.smithi032.stdout:257698037800 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 msgr2=0x7f8fb40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 0x7f8fb40a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005950 tx=0x7f8fc00058e0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 msgr2=0x7f8fcc1990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000c00 tx=0x7f8fbc0047b0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 shutdown_connections 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc107990 0x7f8fcc19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fcc101940 0x7f8fcc1990b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fb40a3cd0 0x7f8fb40a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 --2- 172.21.15.32:0/735497606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fcc102300 0x7f8fcc1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.483+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 >> 172.21.15.32:0/735497606 conn(0x7f8fcc0788f0 msgr2=0x7f8fcc06a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.484+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 shutdown_connections 2024-04-23T18:14:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:00.484+0000 7f8fd195d700 1 -- 172.21.15.32:0/735497606 wait complete. 2024-04-23T18:14:01.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:01 smithi032 ceph-mon[29221]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:14:01.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:01 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/735497606' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:14:01.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:01 smithi040 ceph-mon[32626]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:14:01.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:01 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/735497606' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:14:01.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:01 smithi064 ceph-mon[33062]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-23T18:14:01.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:01 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/735497606' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:14:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.962+0000 7f5678898700 1 -- 172.21.15.32:0/4068234786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f567406b0f0 msgr2=0x7f567410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.962+0000 7f5678898700 1 --2- 172.21.15.32:0/4068234786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f567406b0f0 0x7f567410f900 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f566c00b0a0 tx=0x7f566c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 -- 172.21.15.32:0/4068234786 shutdown_connections 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 --2- 172.21.15.32:0/4068234786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f567406b0f0 0x7f567410f900 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 --2- 172.21.15.32:0/4068234786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5674108a30 0x7f5674108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 --2- 172.21.15.32:0/4068234786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 0x7f567406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 -- 172.21.15.32:0/4068234786 >> 172.21.15.32:0/4068234786 conn(0x7f56740788f0 msgr2=0x7f5674078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 -- 172.21.15.32:0/4068234786 shutdown_connections 2024-04-23T18:14:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.963+0000 7f5678898700 1 -- 172.21.15.32:0/4068234786 wait complete. 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.964+0000 7f5678898700 1 Processor -- start 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.964+0000 7f5678898700 1 -- start start 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.964+0000 7f5678898700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 0x7f5674101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.964+0000 7f5678898700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f567406b0f0 0x7f5674101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.964+0000 7f5678898700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5678898700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5674112c70 con 0x7f5674108a30 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5678898700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5674112af0 con 0x7f567406a750 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5678898700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5674112df0 con 0x7f567406b0f0 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38164/0 (socket says 172.21.15.32:38164) 2024-04-23T18:14:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 -- 172.21.15.32:0/3625168435 learned_addr learned my addr 172.21.15.32:0/3625168435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5671d9b700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f567406b0f0 0x7f5674101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f567259c700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 0x7f5674101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 -- 172.21.15.32:0/3625168435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f567406b0f0 msgr2=0x7f5674101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f567406b0f0 0x7f5674101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 -- 172.21.15.32:0/3625168435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 msgr2=0x7f5674101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 0x7f5674101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 -- 172.21.15.32:0/3625168435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56741a74c0 con 0x7f5674108a30 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.965+0000 7f5672d9d700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f566c00b830 tx=0x7f566c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.966+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c026070 con 0x7f5674108a30 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.966+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f566c0079e0 con 0x7f5674108a30 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.966+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5674198d20 con 0x7f5674108a30 2024-04-23T18:14:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.966+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c004560 con 0x7f5674108a30 2024-04-23T18:14:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.966+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5674199260 con 0x7f5674108a30 2024-04-23T18:14:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.967+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5674069730 con 0x7f5674108a30 2024-04-23T18:14:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.968+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f566c004700 con 0x7f5674108a30 2024-04-23T18:14:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.969+0000 7f566b7fe700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 0x7f56600a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.969+0000 7f567259c700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 0x7f56600a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.969+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f566c0d9b60 con 0x7f5674108a30 2024-04-23T18:14:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.970+0000 7f567259c700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 0x7f56600a62e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f565c005950 tx=0x7f565c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:01.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:01.977+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f566c011390 con 0x7f5674108a30 2024-04-23T18:14:02.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.194+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5674199620 con 0x7f5674108a30 2024-04-23T18:14:02.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.194+0000 7f566b7fe700 1 -- 172.21.15.32:0/3625168435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f566c0a6630 con 0x7f5674108a30 2024-04-23T18:14:02.190 INFO:teuthology.orchestra.run.smithi032.stdout:107374182465 2024-04-23T18:14:02.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 msgr2=0x7f56600a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 0x7f56600a62e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f565c005950 tx=0x7f565c0058e0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 msgr2=0x7f5674102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f566c00b830 tx=0x7f566c009c20 comp rx=0 tx=0).stop 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 shutdown_connections 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.196+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5674108a30 0x7f5674102490 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56600a3e20 0x7f56600a62e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567406a750 0x7f5674101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 --2- 172.21.15.32:0/3625168435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f567406b0f0 0x7f5674101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 >> 172.21.15.32:0/3625168435 conn(0x7f56740788f0 msgr2=0x7f56740ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:02.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 shutdown_connections 2024-04-23T18:14:02.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.197+0000 7f5678898700 1 -- 172.21.15.32:0/3625168435 wait complete. 2024-04-23T18:14:02.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3625168435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:14:02.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3625168435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:14:02.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3625168435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:14:02.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 -- 172.21.15.32:0/2875418731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806b0f0 msgr2=0x7f580810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 --2- 172.21.15.32:0/2875418731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806b0f0 0x7f580810f900 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009990 tx=0x7f57f8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 -- 172.21.15.32:0/2875418731 shutdown_connections 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 --2- 172.21.15.32:0/2875418731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806b0f0 0x7f580810f900 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 --2- 172.21.15.32:0/2875418731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5808108a30 0x7f5808108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 --2- 172.21.15.32:0/2875418731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f580806a750 0x7f580806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 -- 172.21.15.32:0/2875418731 >> 172.21.15.32:0/2875418731 conn(0x7f58080788f0 msgr2=0x7f5808078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 -- 172.21.15.32:0/2875418731 shutdown_connections 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 -- 172.21.15.32:0/2875418731 wait complete. 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.769+0000 7f580d270700 1 Processor -- start 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 -- start start 2024-04-23T18:14:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580806b0f0 0x7f5808101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5808108a30 0x7f5808102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5808112c60 con 0x7f580806a750 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5808112ae0 con 0x7f580806b0f0 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f580d270700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5808112de0 con 0x7f5808108a30 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38180/0 (socket says 172.21.15.32:38180) 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 -- 172.21.15.32:0/2468009863 learned_addr learned my addr 172.21.15.32:0/2468009863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f58077fe700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5808108a30 0x7f5808102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f58067fc700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580806b0f0 0x7f5808101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 -- 172.21.15.32:0/2468009863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5808108a30 msgr2=0x7f5808102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5808108a30 0x7f5808102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 -- 172.21.15.32:0/2468009863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580806b0f0 msgr2=0x7f5808101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580806b0f0 0x7f5808101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.770+0000 7f5806ffd700 1 -- 172.21.15.32:0/2468009863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58081a76a0 con 0x7f580806a750 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f5806ffd700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f58000060b0 tx=0x7f580000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580001a070 con 0x7f580806a750 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f580000dd20 con 0x7f580806a750 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58081a7990 con 0x7f580806a750 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800014600 con 0x7f580806a750 2024-04-23T18:14:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.771+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58081a7ed0 con 0x7f580806a750 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.772+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5808069730 con 0x7f580806a750 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.773+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f580001d070 con 0x7f580806a750 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.774+0000 7f57fffff700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 0x7f57f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.774+0000 7f58067fc700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 0x7f57f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.774+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f58000cfd40 con 0x7f580806a750 2024-04-23T18:14:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.774+0000 7f58067fc700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 0x7f57f40a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5808078ed0 tx=0x7f57f000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.778+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5800010070 con 0x7f580806a750 2024-04-23T18:14:02.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.988+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f580802df80 con 0x7f580806a750 2024-04-23T18:14:02.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.989+0000 7f57fffff700 1 -- 172.21.15.32:0/2468009863 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f580009c590 con 0x7f580806a750 2024-04-23T18:14:02.983 INFO:teuthology.orchestra.run.smithi032.stdout:201863462963 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 msgr2=0x7f57f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 0x7f57f40a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5808078ed0 tx=0x7f57f000b3c0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 msgr2=0x7f5808101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f58000060b0 tx=0x7f580000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 shutdown_connections 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5808108a30 0x7f5808102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f580806a750 0x7f5808101a10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f57f40a3cd0 0x7f57f40a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 --2- 172.21.15.32:0/2468009863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580806b0f0 0x7f5808101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 >> 172.21.15.32:0/2468009863 conn(0x7f58080788f0 msgr2=0x7f58080ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 shutdown_connections 2024-04-23T18:14:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:02.991+0000 7f580d270700 1 -- 172.21.15.32:0/2468009863 wait complete. 2024-04-23T18:14:03.180 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182462 got 107374182465 for osd.3 2024-04-23T18:14:03.180 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:03.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:03 smithi032 ceph-mon[29221]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:03.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2468009863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:14:03.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:03 smithi040 ceph-mon[32626]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:03.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2468009863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:14:03.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:03 smithi064 ceph-mon[33062]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:03.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2468009863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:14:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- 172.21.15.32:0/617903810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006afe0 msgr2=0x7faa3006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 --2- 172.21.15.32:0/617903810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006afe0 0x7faa3006b440 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009990 tx=0x7faa1c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- 172.21.15.32:0/617903810 shutdown_connections 2024-04-23T18:14:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 --2- 172.21.15.32:0/617903810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006b980 0x7faa3010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 --2- 172.21.15.32:0/617903810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 0x7faa30108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 --2- 172.21.15.32:0/617903810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006afe0 0x7faa3006b440 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- 172.21.15.32:0/617903810 >> 172.21.15.32:0/617903810 conn(0x7faa300788b0 msgr2=0x7faa30078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- 172.21.15.32:0/617903810 shutdown_connections 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- 172.21.15.32:0/617903810 wait complete. 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 Processor -- start 2024-04-23T18:14:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.930+0000 7faa36fb9700 1 -- start start 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006afe0 0x7faa30101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 0x7faa30102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112be0 con 0x7faa3006b980 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112a60 con 0x7faa3006afe0 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112d60 con 0x7faa30108a20 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38192/0 (socket says 172.21.15.32:38192) 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 -- 172.21.15.32:0/4095158261 learned_addr learned my addr 172.21.15.32:0/4095158261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa34d55700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006afe0 0x7faa30101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa35556700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 0x7faa30102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 -- 172.21.15.32:0/4095158261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 msgr2=0x7faa30102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 0x7faa30102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 -- 172.21.15.32:0/4095158261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006afe0 msgr2=0x7faa30101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006afe0 0x7faa30101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 -- 172.21.15.32:0/4095158261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a7680 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2ffff700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c027070 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c003f20 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a7910 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.931+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c022cf0 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.932+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a7dc0 con 0x7faa3006b980 2024-04-23T18:14:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.933+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa30069730 con 0x7faa3006b980 2024-04-23T18:14:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.934+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa1c02a070 con 0x7faa3006b980 2024-04-23T18:14:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.934+0000 7faa2dffb700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 0x7faa200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.935+0000 7faa34d55700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 0x7faa200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.935+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7faa1c0d1690 con 0x7faa3006b980 2024-04-23T18:14:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.935+0000 7faa34d55700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 0x7faa200a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faa240045b0 tx=0x7faa2400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:03.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:03.938+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa1c013070 con 0x7faa3006b980 2024-04-23T18:14:04.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.151+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7faa3002e140 con 0x7faa3006b980 2024-04-23T18:14:04.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.152+0000 7faa2dffb700 1 -- 172.21.15.32:0/4095158261 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faa1c09df70 con 0x7faa3006b980 2024-04-23T18:14:04.146 INFO:teuthology.orchestra.run.smithi032.stdout:292057776164 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 msgr2=0x7faa200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 0x7faa200a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faa240045b0 tx=0x7faa2400b3c0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 msgr2=0x7faa30101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0047b0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 shutdown_connections 2024-04-23T18:14:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa30108a20 0x7faa30102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa200a3d50 0x7faa200a6210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa3006afe0 0x7faa30101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 --2- 172.21.15.32:0/4095158261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa3006b980 0x7faa30101f40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 >> 172.21.15.32:0/4095158261 conn(0x7faa300788b0 msgr2=0x7faa300ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 shutdown_connections 2024-04-23T18:14:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:04.155+0000 7faa36fb9700 1 -- 172.21.15.32:0/4095158261 wait complete. 2024-04-23T18:14:04.345 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037797 got 257698037800 for osd.8 2024-04-23T18:14:04.345 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:04.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:04 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4095158261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:14:04.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:04 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4095158261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:14:04.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:04 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4095158261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:14:05.247 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776159 got 292057776164 for osd.9 2024-04-23T18:14:05.247 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:05.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:05 smithi032 ceph-mon[29221]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:05.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:05 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:05.573 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:05 smithi064 ceph-mon[33062]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:05.573 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:05 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:05.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:05 smithi040 ceph-mon[32626]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 145 B/s, 1 keys/s, 1 objects/s recovering 2024-04-23T18:14:05.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:05 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- 172.21.15.32:0/4050703728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 msgr2=0x7faf6006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 --2- 172.21.15.32:0/4050703728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf6006b5f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf48009990 tx=0x7faf48009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- 172.21.15.32:0/4050703728 shutdown_connections 2024-04-23T18:14:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 --2- 172.21.15.32:0/4050703728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 0x7faf6010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 --2- 172.21.15.32:0/4050703728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 0x7faf60108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 --2- 172.21.15.32:0/4050703728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf6006b5f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- 172.21.15.32:0/4050703728 >> 172.21.15.32:0/4050703728 conn(0x7faf600777b0 msgr2=0x7faf60077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- 172.21.15.32:0/4050703728 shutdown_connections 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- 172.21.15.32:0/4050703728 wait complete. 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 Processor -- start 2024-04-23T18:14:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.333+0000 7faf665d4700 1 -- start start 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 0x7faf60199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 0x7faf6019d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf60112c30 con 0x7faf6006b170 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faf60112ab0 con 0x7faf60108a70 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf665d4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faf60112db0 con 0x7faf6006bb30 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38202/0 (socket says 172.21.15.32:38202) 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 -- 172.21.15.32:0/2625996657 learned_addr learned my addr 172.21.15.32:0/2625996657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf64b71700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 0x7faf6019d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5f7fe700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 0x7faf60199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 -- 172.21.15.32:0/2625996657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 msgr2=0x7faf60199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 0x7faf60199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 -- 172.21.15.32:0/2625996657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 msgr2=0x7faf6019d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 0x7faf6019d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.334+0000 7faf5ffff700 1 -- 172.21.15.32:0/2625996657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6019e0f0 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf5ffff700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf5000d950 tx=0x7faf5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50016070 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5000fd40 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6019e3e0 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5000e3b0 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.335+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf601a5c60 con 0x7faf6006b170 2024-04-23T18:14:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.336+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf601041a0 con 0x7faf6006b170 2024-04-23T18:14:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.337+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faf50019070 con 0x7faf6006b170 2024-04-23T18:14:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.337+0000 7faf5d7fa700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 0x7faf4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.338+0000 7faf5f7fe700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 0x7faf4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.338+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7faf500cff90 con 0x7faf6006b170 2024-04-23T18:14:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.338+0000 7faf5f7fe700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 0x7faf4c0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf60077e90 tx=0x7faf48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.342+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf5009c840 con 0x7faf6006b170 2024-04-23T18:14:06.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.557+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7faf6002df80 con 0x7faf6006b170 2024-04-23T18:14:06.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.557+0000 7faf5d7fa700 1 -- 172.21.15.32:0/2625996657 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faf5009c140 con 0x7faf6006b170 2024-04-23T18:14:06.552 INFO:teuthology.orchestra.run.smithi032.stdout:141733920830 2024-04-23T18:14:06.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.559+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 msgr2=0x7faf4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.559+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 0x7faf4c0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf60077e90 tx=0x7faf48005960 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.559+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 msgr2=0x7faf601990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.559+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf5000d950 tx=0x7faf5000dc60 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 shutdown_connections 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faf4c0a3cd0 0x7faf4c0a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf60108a70 0x7faf6019d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf6006b170 0x7faf601990e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 --2- 172.21.15.32:0/2625996657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf6006bb30 0x7faf60199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 >> 172.21.15.32:0/2625996657 conn(0x7faf600777b0 msgr2=0x7faf60100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 shutdown_connections 2024-04-23T18:14:06.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:06.560+0000 7faf665d4700 1 -- 172.21.15.32:0/2625996657 wait complete. 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- 172.21.15.32:0/1143553061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 msgr2=0x7f8ad4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/1143553061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad4102f10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009990 tx=0x7f8ac8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- 172.21.15.32:0/1143553061 shutdown_connections 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/1143553061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4103450 0x7f8ad410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/1143553061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4108a50 0x7f8ad4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/1143553061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad4102f10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- 172.21.15.32:0/1143553061 >> 172.21.15.32:0/1143553061 conn(0x7f8ad4079ba0 msgr2=0x7f8ad4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- 172.21.15.32:0/1143553061 shutdown_connections 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- 172.21.15.32:0/1143553061 wait complete. 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 Processor -- start 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.008+0000 7f8ada7bc700 1 -- start start 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4103450 0x7f8ad4199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4108a50 0x7f8ad419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4112c50 con 0x7f8ad4102a90 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4112ad0 con 0x7f8ad4108a50 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ada7bc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4112dd0 con 0x7f8ad4103450 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57198/0 (socket says 172.21.15.32:57198) 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 -- 172.21.15.32:0/700068673 learned_addr learned my addr 172.21.15.32:0/700068673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad37fe700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4103450 0x7f8ad4199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad8d59700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4108a50 0x7f8ad419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 -- 172.21.15.32:0/700068673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4103450 msgr2=0x7f8ad4199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4103450 0x7f8ad4199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 -- 172.21.15.32:0/700068673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4108a50 msgr2=0x7f8ad419da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4108a50 0x7f8ad419da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 -- 172.21.15.32:0/700068673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad419e1d0 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.009+0000 7f8ad3fff700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400e9c0 tx=0x7f8ac400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.010+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac400cbd0 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.010+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad419e4c0 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.010+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ac400ee60 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.010+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40055c0 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.010+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad41a5d40 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.011+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad4069730 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.012+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ac4004750 con 0x7f8ad4102a90 2024-04-23T18:14:07.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.013+0000 7f8ad17fa700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 0x7f8abc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.013+0000 7f8ad37fe700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 0x7f8abc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.013+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8ac40d09e0 con 0x7f8ad4102a90 2024-04-23T18:14:07.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.013+0000 7f8ad37fe700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 0x7f8abc0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ad407a2e0 tx=0x7f8ac8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:07.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.018+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ac4053ff0 con 0x7f8ad4102a90 2024-04-23T18:14:07.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.231+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8ad419a520 con 0x7f8ad4102a90 2024-04-23T18:14:07.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.231+0000 7f8ad17fa700 1 -- 172.21.15.32:0/700068673 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8ac409d230 con 0x7f8ad4102a90 2024-04-23T18:14:07.225 INFO:teuthology.orchestra.run.smithi032.stdout:77309411400 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 msgr2=0x7f8abc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 0x7f8abc0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ad407a2e0 tx=0x7f8ac8005fd0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 msgr2=0x7f8ad41991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400e9c0 tx=0x7f8ac400ecd0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 shutdown_connections 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8abc0a3cd0 0x7f8abc0a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ad4108a50 0x7f8ad419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4102a90 0x7f8ad41991c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 --2- 172.21.15.32:0/700068673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad4103450 0x7f8ad4199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.233+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 >> 172.21.15.32:0/700068673 conn(0x7f8ad4079ba0 msgr2=0x7f8ad4101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.234+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 shutdown_connections 2024-04-23T18:14:07.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:07.234+0000 7f8ada7bc700 1 -- 172.21.15.32:0/700068673 wait complete. 2024-04-23T18:14:07.468 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:14:07.468 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2625996657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:14:07.468 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.469 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.469 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.469 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:07 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.517 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:14:07.517 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2625996657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:14:07.517 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.518 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:07 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2625996657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:07.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:07 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:08.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:08 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/700068673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:14:08.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:08 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/700068673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:14:08.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:08 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/700068673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:14:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:08.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462959 got 201863462963 for osd.6 2024-04-23T18:14:08.911 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:09.469 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:09 smithi032 ceph-mon[29221]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 106 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:09 smithi040 ceph-mon[32626]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 106 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:09.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:09 smithi064 ceph-mon[33062]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 106 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:10.466 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411395 got 77309411400 for osd.2 2024-04-23T18:14:10.467 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:11.299 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920826 got 141733920830 for osd.4 2024-04-23T18:14:11.299 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:11.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:11 smithi032 ceph-mon[29221]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 -- 172.21.15.32:0/641852458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34102300 msgr2=0x7faa3410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 --2- 172.21.15.32:0/641852458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34102300 0x7faa3410e7f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faa2c00b0a0 tx=0x7faa2c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 -- 172.21.15.32:0/641852458 shutdown_connections 2024-04-23T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 --2- 172.21.15.32:0/641852458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34102300 0x7faa3410e7f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 --2- 172.21.15.32:0/641852458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34107990 0x7faa34107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 --2- 172.21.15.32:0/641852458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34101940 0x7faa34101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 -- 172.21.15.32:0/641852458 >> 172.21.15.32:0/641852458 conn(0x7faa340788f0 msgr2=0x7faa34078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 -- 172.21.15.32:0/641852458 shutdown_connections 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.519+0000 7faa3a953700 1 -- 172.21.15.32:0/641852458 wait complete. 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 Processor -- start 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 -- start start 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34102300 0x7faa341995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34107990 0x7faa3419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa34111ae0 con 0x7faa34101940 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa34111960 con 0x7faa34102300 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.520+0000 7faa3a953700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa34111c60 con 0x7faa34107990 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57214/0 (socket says 172.21.15.32:57214) 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 -- 172.21.15.32:0/3903166189 learned_addr learned my addr 172.21.15.32:0/3903166189 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa38ef0700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34107990 0x7faa3419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa337fe700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34102300 0x7faa341995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 -- 172.21.15.32:0/3903166189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34107990 msgr2=0x7faa3419d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34107990 0x7faa3419d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 -- 172.21.15.32:0/3903166189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34102300 msgr2=0x7faa341995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34102300 0x7faa341995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 -- 172.21.15.32:0/3903166189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3419e070 con 0x7faa34101940 2024-04-23T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa33fff700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faa2400d950 tx=0x7faa2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24016070 con 0x7faa34101940 2024-04-23T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3419e360 con 0x7faa34101940 2024-04-23T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2400fd40 con 0x7faa34101940 2024-04-23T18:14:11.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2400e3b0 con 0x7faa34101940 2024-04-23T18:14:11.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.521+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa341a5be0 con 0x7faa34101940 2024-04-23T18:14:11.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.522+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa34069730 con 0x7faa34101940 2024-04-23T18:14:11.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.523+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa24019070 con 0x7faa34101940 2024-04-23T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.524+0000 7faa317fa700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 0x7faa1c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.524+0000 7faa337fe700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 0x7faa1c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.524+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7faa240cffa0 con 0x7faa34101940 2024-04-23T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.525+0000 7faa337fe700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 0x7faa1c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa34079030 tx=0x7faa2800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.528+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa2409c7f0 con 0x7faa34101940 2024-04-23T18:14:11.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:11 smithi040 ceph-mon[32626]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:11.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:11 smithi064 ceph-mon[33062]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.742+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7faa3419a3c0 con 0x7faa34101940 2024-04-23T18:14:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.742+0000 7faa317fa700 1 -- 172.21.15.32:0/3903166189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faa2409c0f0 con 0x7faa34101940 2024-04-23T18:14:11.737 INFO:teuthology.orchestra.run.smithi032.stdout:34359738451 2024-04-23T18:14:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 msgr2=0x7faa1c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 0x7faa1c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa34079030 tx=0x7faa2800b410 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 msgr2=0x7faa34199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faa2400d950 tx=0x7faa2400dc60 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 shutdown_connections 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa34107990 0x7faa3419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa34101940 0x7faa34199060 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7faa1c0a3cd0 0x7faa1c0a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 --2- 172.21.15.32:0/3903166189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa34102300 0x7faa341995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 >> 172.21.15.32:0/3903166189 conn(0x7faa340788f0 msgr2=0x7faa3406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 shutdown_connections 2024-04-23T18:14:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:11.744+0000 7faa3a953700 1 -- 172.21.15.32:0/3903166189 wait complete. 2024-04-23T18:14:12.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3903166189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:14:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:12.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:12 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3903166189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:14:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:12 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3903166189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:14:13.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:13 smithi032 ceph-mon[29221]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:13.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738451 for osd.0 2024-04-23T18:14:13.448 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:13.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:13 smithi040 ceph-mon[32626]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:13.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:13 smithi064 ceph-mon[33062]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:14:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 -- 172.21.15.32:0/2363661251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54107920 msgr2=0x7fda54107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 --2- 172.21.15.32:0/2363661251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54107920 0x7fda54107d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fda50009990 tx=0x7fda50009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 -- 172.21.15.32:0/2363661251 shutdown_connections 2024-04-23T18:14:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 --2- 172.21.15.32:0/2363661251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54102330 0x7fda5410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 --2- 172.21.15.32:0/2363661251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54107920 0x7fda54107d00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 --2- 172.21.15.32:0/2363661251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54101970 0x7fda54101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 -- 172.21.15.32:0/2363661251 >> 172.21.15.32:0/2363661251 conn(0x7fda540779b0 msgr2=0x7fda54077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 -- 172.21.15.32:0/2363661251 shutdown_connections 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.150+0000 7fda5c429700 1 -- 172.21.15.32:0/2363661251 wait complete. 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 Processor -- start 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 -- start start 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54101970 0x7fda54197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54107920 0x7fda541987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda54111b30 con 0x7fda54102330 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda541119b0 con 0x7fda54101970 2024-04-23T18:14:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5c429700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fda54111cb0 con 0x7fda54107920 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57230/0 (socket says 172.21.15.32:57230) 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 -- 172.21.15.32:0/1311166890 learned_addr learned my addr 172.21.15.32:0/1311166890 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5a9c6700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54107920 0x7fda541987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda5a1c5700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54101970 0x7fda54197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 -- 172.21.15.32:0/1311166890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54107920 msgr2=0x7fda541987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54107920 0x7fda541987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.151+0000 7fda599c4700 1 -- 172.21.15.32:0/1311166890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54101970 msgr2=0x7fda54197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda599c4700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54101970 0x7fda54197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda599c4700 1 -- 172.21.15.32:0/1311166890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda541a7660 con 0x7fda54102330 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda599c4700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c970 tx=0x7fda4400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44007aa0 con 0x7fda54102330 2024-04-23T18:14:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda44004ce0 con 0x7fda54102330 2024-04-23T18:14:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda541a7950 con 0x7fda54102330 2024-04-23T18:14:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44005510 con 0x7fda54102330 2024-04-23T18:14:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.152+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda541a7e90 con 0x7fda54102330 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.153+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda54109f20 con 0x7fda54102330 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.154+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fda44007c40 con 0x7fda54102330 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.155+0000 7fda4b7fe700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 0x7fda400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.155+0000 7fda5a1c5700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 0x7fda400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.155+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fda440d04c0 con 0x7fda54102330 2024-04-23T18:14:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.155+0000 7fda5a1c5700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 0x7fda400a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda50006010 tx=0x7fda5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:15.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.158+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda4409ce90 con 0x7fda54102330 2024-04-23T18:14:15.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:15 smithi032 ceph-mon[29221]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:15.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.370+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fda54109900 con 0x7fda54102330 2024-04-23T18:14:15.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.371+0000 7fda4b7fe700 1 -- 172.21.15.32:0/1311166890 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fda4409c790 con 0x7fda54102330 2024-04-23T18:14:15.365 INFO:teuthology.orchestra.run.smithi032.stdout:167503724603 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 msgr2=0x7fda400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 0x7fda400a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda50006010 tx=0x7fda5001a040 comp rx=0 tx=0).stop 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 msgr2=0x7fda541982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c970 tx=0x7fda4400cc80 comp rx=0 tx=0).stop 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 shutdown_connections 2024-04-23T18:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda54107920 0x7fda541987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fda400a3d50 0x7fda400a6210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda54101970 0x7fda54197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 --2- 172.21.15.32:0/1311166890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda54102330 0x7fda541982a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 >> 172.21.15.32:0/1311166890 conn(0x7fda540779b0 msgr2=0x7fda540ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 shutdown_connections 2024-04-23T18:14:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.373+0000 7fda5c429700 1 -- 172.21.15.32:0/1311166890 wait complete. 2024-04-23T18:14:15.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 -- 172.21.15.32:0/792282633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 msgr2=0x7f1594102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 --2- 172.21.15.32:0/792282633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f1594102f70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f15840099c0 tx=0x7f1584009cd0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 -- 172.21.15.32:0/792282633 shutdown_connections 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 --2- 172.21.15.32:0/792282633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 0x7f159410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 --2- 172.21.15.32:0/792282633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 0x7f1594107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 --2- 172.21.15.32:0/792282633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f1594102f70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 -- 172.21.15.32:0/792282633 >> 172.21.15.32:0/792282633 conn(0x7f1594079ba0 msgr2=0x7f1594079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 -- 172.21.15.32:0/792282633 shutdown_connections 2024-04-23T18:14:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.390+0000 7f159b3a0700 1 -- 172.21.15.32:0/792282633 wait complete. 2024-04-23T18:14:15.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 Processor -- start 2024-04-23T18:14:15.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 -- start start 2024-04-23T18:14:15.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 0x7f15941996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 0x7f159419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1594112cb0 con 0x7f1594102af0 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1594112b30 con 0x7f15941034b0 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.391+0000 7f159b3a0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1594112e30 con 0x7f1594107940 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57244/0 (socket says 172.21.15.32:57244) 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 -- 172.21.15.32:0/3084744538 learned_addr learned my addr 172.21.15.32:0/3084744538 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159993d700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 0x7f159419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159893b700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 0x7f15941996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 -- 172.21.15.32:0/3084744538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 msgr2=0x7f159419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 0x7f159419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 -- 172.21.15.32:0/3084744538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 msgr2=0x7f15941996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 0x7f15941996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 -- 172.21.15.32:0/3084744538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159419e1b0 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159913c700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f159000c970 tx=0x7f159000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1590007aa0 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159419e4a0 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1590004ce0 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1590005510 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.392+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15941a5d20 con 0x7f1594102af0 2024-04-23T18:14:15.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.393+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1594069730 con 0x7f1594102af0 2024-04-23T18:14:15.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.395+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1590007c40 con 0x7f1594102af0 2024-04-23T18:14:15.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.396+0000 7f158a7fc700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 0x7f15800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:15.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.396+0000 7f159893b700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 0x7f15800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:15.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.396+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f15900d0520 con 0x7f1594102af0 2024-04-23T18:14:15.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.397+0000 7f159893b700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 0x7f15800a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f159407a180 tx=0x7f158401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:15.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.400+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f159009cef0 con 0x7f1594102af0 2024-04-23T18:14:15.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:15 smithi040 ceph-mon[32626]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:15.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.628+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f159419a530 con 0x7f1594102af0 2024-04-23T18:14:15.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.629+0000 7f158a7fc700 1 -- 172.21.15.32:0/3084744538 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f159419a530 con 0x7f1594102af0 2024-04-23T18:14:15.623 INFO:teuthology.orchestra.run.smithi032.stdout:313532612640 2024-04-23T18:14:15.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 msgr2=0x7f15800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 0x7f15800a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f159407a180 tx=0x7f158401a040 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 msgr2=0x7f15941991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f159000c970 tx=0x7f159000cc80 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 shutdown_connections 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1594107940 0x7f159419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1594102af0 0x7f15941991a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f15800a3cd0 0x7f15800a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 --2- 172.21.15.32:0/3084744538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15941034b0 0x7f15941996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.631+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 >> 172.21.15.32:0/3084744538 conn(0x7f1594079ba0 msgr2=0x7f1594100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.632+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 shutdown_connections 2024-04-23T18:14:15.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.632+0000 7f159b3a0700 1 -- 172.21.15.32:0/3084744538 wait complete. 2024-04-23T18:14:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:15 smithi064 ceph-mon[33062]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:16.358 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612640 for osd.10 2024-04-23T18:14:16.358 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:16.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1311166890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:14:16.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3084744538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:14:16.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3510171979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:14:16.619 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1311166890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:14:16.619 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3084744538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:14:16.619 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3510171979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:14:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1311166890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:14:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3084744538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:14:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3510171979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:14:17.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724597 got 167503724603 for osd.5 2024-04-23T18:14:17.182 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/2490587911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 msgr2=0x7f0b8810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/2490587911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b8810f900 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000b0a0 tx=0x7f0b8000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/2490587911 shutdown_connections 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/2490587911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b8810f900 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/2490587911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 0x7f0b88108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/2490587911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 0x7f0b8806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/2490587911 >> 172.21.15.32:0/2490587911 conn(0x7f0b880788f0 msgr2=0x7f0b88078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/2490587911 shutdown_connections 2024-04-23T18:14:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.591+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/2490587911 wait complete. 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 Processor -- start 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 -- start start 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 0x7f0b88199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 0x7f0b8819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0b88112bd0 con 0x7f0b8806b0f0 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b88112a50 con 0x7f0b8806a750 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b8e1a0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b88112d50 con 0x7f0b88108a30 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b86ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b86ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57268/0 (socket says 172.21.15.32:57268) 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b86ffd700 1 -- 172.21.15.32:0/3510171979 learned_addr learned my addr 172.21.15.32:0/3510171979 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b877fe700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 0x7f0b88199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.592+0000 7f0b87fff700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 0x7f0b8819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 -- 172.21.15.32:0/3510171979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 msgr2=0x7f0b8819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 0x7f0b8819d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 -- 172.21.15.32:0/3510171979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 msgr2=0x7f0b88199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 0x7f0b88199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 -- 172.21.15.32:0/3510171979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8819e110 con 0x7f0b8806b0f0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b86ffd700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00e9a0 tx=0x7f0b7c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c00cbd0 con 0x7f0b8806b0f0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8819e400 con 0x7f0b8806b0f0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b7c00cd70 con 0x7f0b8806b0f0 2024-04-23T18:14:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c018680 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stdout:223338299440 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.593+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b881a5c80 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.594+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b88069730 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.595+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0b7c010970 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.596+0000 7f0b84ff9700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 0x7f0b700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.596+0000 7f0b877fe700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 0x7f0b700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.596+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0b7c0d0860 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.596+0000 7f0b877fe700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 0x7f0b700a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009bd0 tx=0x7f0b78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.601+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b7c09d110 con 0x7f0b8806b0f0 2024-04-23T18:14:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.832+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f0b8802df80 con 0x7f0b8806b0f0 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.832+0000 7f0b84ff9700 1 -- 172.21.15.32:0/3510171979 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0b7c09ca10 con 0x7f0b8806b0f0 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 msgr2=0x7f0b700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 0x7f0b700a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009bd0 tx=0x7f0b78009260 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 msgr2=0x7f0b88199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00e9a0 tx=0x7f0b7c00ed60 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 shutdown_connections 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b88108a30 0x7f0b8819d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0b700a3cd0 0x7f0b700a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b8806a750 0x7f0b88199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 --2- 172.21.15.32:0/3510171979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8806b0f0 0x7f0b88199640 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 >> 172.21.15.32:0/3510171979 conn(0x7f0b880788f0 msgr2=0x7f0b880ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 shutdown_connections 2024-04-23T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:15.834+0000 7f0b8e1a0700 1 -- 172.21.15.32:0/3510171979 wait complete. 2024-04-23T18:14:17.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:17 smithi032 ceph-mon[29221]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:17.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:17 smithi040 ceph-mon[32626]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:17.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:17 smithi064 ceph-mon[33062]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:17.931 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299433 got 223338299440 for osd.7 2024-04-23T18:14:17.931 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 -- 172.21.15.32:0/402440068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 msgr2=0x7f44e010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 --2- 172.21.15.32:0/402440068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e010f900 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f44d800b0a0 tx=0x7f44d800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 -- 172.21.15.32:0/402440068 shutdown_connections 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 --2- 172.21.15.32:0/402440068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e010f900 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 --2- 172.21.15.32:0/402440068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 0x7f44e0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 --2- 172.21.15.32:0/402440068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 0x7f44e006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.623+0000 7f44e5610700 1 -- 172.21.15.32:0/402440068 >> 172.21.15.32:0/402440068 conn(0x7f44e00788f0 msgr2=0x7f44e0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.624+0000 7f44e5610700 1 -- 172.21.15.32:0/402440068 shutdown_connections 2024-04-23T18:14:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.624+0000 7f44e5610700 1 -- 172.21.15.32:0/402440068 wait complete. 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.624+0000 7f44e5610700 1 Processor -- start 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.624+0000 7f44e5610700 1 -- start start 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.624+0000 7f44e5610700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 0x7f44e0101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 0x7f44e01024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0112cb0 con 0x7f44e006bb10 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0112b30 con 0x7f44e0108a30 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0112e30 con 0x7f44e006b170 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33424/0 (socket says 172.21.15.32:33424) 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 -- 172.21.15.32:0/2267145953 learned_addr learned my addr 172.21.15.32:0/2267145953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44deffd700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 0x7f44e0101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44df7fe700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 0x7f44e01024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 -- 172.21.15.32:0/2267145953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 msgr2=0x7f44e0101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 0x7f44e0101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 -- 172.21.15.32:0/2267145953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 msgr2=0x7f44e01024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 0x7f44e01024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 -- 172.21.15.32:0/2267145953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e01a7790 con 0x7f44e006bb10 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44de7fc700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f44d400b530 tx=0x7f44d400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4015020 con 0x7f44e006bb10 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d4011ce0 con 0x7f44e006bb10 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e01a7a80 con 0x7f44e006bb10 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4010e40 con 0x7f44e006bb10 2024-04-23T18:14:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.625+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e01a7f70 con 0x7f44e006bb10 2024-04-23T18:14:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.626+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e0069730 con 0x7f44e006bb10 2024-04-23T18:14:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.627+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f44d4018020 con 0x7f44e006bb10 2024-04-23T18:14:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.628+0000 7f44c7fff700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 0x7f44c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.628+0000 7f44deffd700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 0x7f44c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.628+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f44d40cfce0 con 0x7f44e006bb10 2024-04-23T18:14:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.629+0000 7f44deffd700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 0x7f44c80a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009a10 tx=0x7f44d0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.633+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44d401f020 con 0x7f44e006bb10 2024-04-23T18:14:18.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.844+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f44e002df80 con 0x7f44e006bb10 2024-04-23T18:14:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.845+0000 7f44c7fff700 1 -- 172.21.15.32:0/2267145953 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f44d409c6a0 con 0x7f44e006bb10 2024-04-23T18:14:18.839 INFO:teuthology.orchestra.run.smithi032.stdout:347892351003 2024-04-23T18:14:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 msgr2=0x7f44c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 0x7f44c80a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009a10 tx=0x7f44d0009190 comp rx=0 tx=0).stop 2024-04-23T18:14:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 msgr2=0x7f44e0101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f44d400b530 tx=0x7f44d400b840 comp rx=0 tx=0).stop 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 shutdown_connections 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f44c80a3cd0 0x7f44c80a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e0108a30 0x7f44e01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44e006b170 0x7f44e0101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 --2- 172.21.15.32:0/2267145953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e006bb10 0x7f44e0101f80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 >> 172.21.15.32:0/2267145953 conn(0x7f44e00788f0 msgr2=0x7f44e00ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 shutdown_connections 2024-04-23T18:14:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.847+0000 7f44e5610700 1 -- 172.21.15.32:0/2267145953 wait complete. 2024-04-23T18:14:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/3820267111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9406b900 msgr2=0x7fbe9410efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/3820267111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9406b900 0x7fbe9410efa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009990 tx=0x7fbe84009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/3820267111 shutdown_connections 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/3820267111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9410f4e0 0x7fbe941118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/3820267111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 0x7fbe9406b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/3820267111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9406b900 0x7fbe9410efa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/3820267111 >> 172.21.15.32:0/3820267111 conn(0x7fbe940788b0 msgr2=0x7fbe94078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/3820267111 shutdown_connections 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.848+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/3820267111 wait complete. 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 Processor -- start 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 -- start start 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 0x7fbe9419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 0x7fbe9419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94114df0 con 0x7fbe9410f4e0 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94114c70 con 0x7fbe9406afe0 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9c1e4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94114f70 con 0x7fbe9406b900 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9a781700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9a781700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33444/0 (socket says 172.21.15.32:33444) 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9a781700 1 -- 172.21.15.32:0/2172024661 learned_addr learned my addr 172.21.15.32:0/2172024661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe9977f700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 0x7fbe9419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.849+0000 7fbe99f80700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 0x7fbe9419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 -- 172.21.15.32:0/2172024661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 msgr2=0x7fbe9419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 0x7fbe9419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 -- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 msgr2=0x7fbe9419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 -- 172.21.15.32:0/2172024661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe941a02f0 con 0x7fbe9406b900 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9a781700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9977f700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 0x7fbe9419b820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbe840059d0 tx=0x7fbe840047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84027070 con 0x7fbe9406b900 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe941a0580 con 0x7fbe9406b900 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe84022470 con 0x7fbe9406b900 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84017650 con 0x7fbe9406b900 2024-04-23T18:14:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.850+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941a7e60 con 0x7fbe9406b900 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.851+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe94101f50 con 0x7fbe9406b900 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.852+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe8402a070 con 0x7fbe9406b900 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.853+0000 7fbe8affd700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.853+0000 7fbe99f80700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.853+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbe840d16e0 con 0x7fbe9406b900 2024-04-23T18:14:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.854+0000 7fbe99f80700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 0x7fbe800a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbe90005950 tx=0x7fbe9000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:18.857+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe84013070 con 0x7fbe9406b900 2024-04-23T18:14:19.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.070+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbe9402df80 con 0x7fbe9406b900 2024-04-23T18:14:19.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.071+0000 7fbe8affd700 1 -- 172.21.15.32:0/2172024661 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbe8409dfc0 con 0x7fbe9406b900 2024-04-23T18:14:19.065 INFO:teuthology.orchestra.run.smithi032.stdout:55834574928 2024-04-23T18:14:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 msgr2=0x7fbe800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 0x7fbe800a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbe90005950 tx=0x7fbe9000b410 comp rx=0 tx=0).stop 2024-04-23T18:14:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 msgr2=0x7fbe9419b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 0x7fbe9419b820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbe840059d0 tx=0x7fbe840047b0 comp rx=0 tx=0).stop 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 shutdown_connections 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9410f4e0 0x7fbe9419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406afe0 0x7fbe9419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 --2- 172.21.15.32:0/2172024661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe9406b900 0x7fbe9419b820 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 >> 172.21.15.32:0/2172024661 conn(0x7fbe940788b0 msgr2=0x7fbe94110990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 shutdown_connections 2024-04-23T18:14:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:19.073+0000 7fbe9c1e4700 1 -- 172.21.15.32:0/2172024661 wait complete. 2024-04-23T18:14:19.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:19 smithi032 ceph-mon[29221]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:19.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2267145953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:14:19.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2172024661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:14:19.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:19 smithi040 ceph-mon[32626]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:19.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:19 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2267145953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:14:19.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:19 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2172024661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:14:19.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:19 smithi064 ceph-mon[33062]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:19.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:19 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2267145953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:14:19.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:19 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2172024661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:14:20.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574928 for osd.1 2024-04-23T18:14:20.160 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:20.283 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350997 got 347892351003 for osd.11 2024-04-23T18:14:20.283 DEBUG:teuthology.parallel:result is None 2024-04-23T18:14:20.283 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T18:14:20.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:14:21.014 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:21.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:21 smithi032 ceph-mon[29221]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:21.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:21 smithi040 ceph-mon[32626]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:21.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:21 smithi064 ceph-mon[33062]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- 172.21.15.32:0/3596019858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 msgr2=0x7fbd5c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 --2- 172.21.15.32:0/3596019858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c101df0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009990 tx=0x7fbd50009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- 172.21.15.32:0/3596019858 shutdown_connections 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 --2- 172.21.15.32:0/3596019858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c102330 0x7fbd5c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 --2- 172.21.15.32:0/3596019858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c107920 0x7fbd5c107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 --2- 172.21.15.32:0/3596019858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c101df0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- 172.21.15.32:0/3596019858 >> 172.21.15.32:0/3596019858 conn(0x7fbd5c0788f0 msgr2=0x7fbd5c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- 172.21.15.32:0/3596019858 shutdown_connections 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- 172.21.15.32:0/3596019858 wait complete. 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 Processor -- start 2024-04-23T18:14:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.772+0000 7fbd63c98700 1 -- start start 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c102330 0x7fbd5c19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c107920 0x7fbd5c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c111b60 con 0x7fbd5c101970 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c1119e0 con 0x7fbd5c102330 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd63c98700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c111ce0 con 0x7fbd5c107920 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33464/0 (socket says 172.21.15.32:33464) 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 -- 172.21.15.32:0/2961347868 learned_addr learned my addr 172.21.15.32:0/2961347868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61233700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c102330 0x7fbd5c19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd62235700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c107920 0x7fbd5c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 -- 172.21.15.32:0/2961347868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c107920 msgr2=0x7fbd5c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c107920 0x7fbd5c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 -- 172.21.15.32:0/2961347868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c102330 msgr2=0x7fbd5c19b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c102330 0x7fbd5c19b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 -- 172.21.15.32:0/2961347868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5c1a02d0 con 0x7fbd5c101970 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd61a34700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0060b0 tx=0x7fbd4c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.773+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c01a070 con 0x7fbd5c101970 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.774+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5c1a05c0 con 0x7fbd5c101970 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.774+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd4c00dd20 con 0x7fbd5c101970 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.774+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c014600 con 0x7fbd5c101970 2024-04-23T18:14:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.774+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5c1a7e40 con 0x7fbd5c101970 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.775+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd5c069730 con 0x7fbd5c101970 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.776+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbd4c01d070 con 0x7fbd5c101970 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.776+0000 7fbd5affd700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 0x7fbd440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.776+0000 7fbd61233700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 0x7fbd440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.777+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbd4c0cfce0 con 0x7fbd5c101970 2024-04-23T18:14:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.777+0000 7fbd61233700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 0x7fbd440a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000b5c0 tx=0x7fbd50005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:22.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:22.781+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd4c010070 con 0x7fbd5c101970 2024-04-23T18:14:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.004+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbd5c062e80 con 0x7fbd440a3cd0 2024-04-23T18:14:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.013+0000 7fbd5affd700 1 -- 172.21.15.32:0/2961347868 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258866 (secure 0 0 0) 0x7fbd5c062e80 con 0x7fbd440a3cd0 2024-04-23T18:14:23.011 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 msgr2=0x7fbd440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 0x7fbd440a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000b5c0 tx=0x7fbd50005960 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 msgr2=0x7fbd5c19b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0060b0 tx=0x7fbd4c00d3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 shutdown_connections 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd5c107920 0x7fbd5c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c101970 0x7fbd5c19b2c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbd440a3cd0 0x7fbd440a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 --2- 172.21.15.32:0/2961347868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd5c102330 0x7fbd5c19b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.016+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 >> 172.21.15.32:0/2961347868 conn(0x7fbd5c0788f0 msgr2=0x7fbd5c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.017+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 shutdown_connections 2024-04-23T18:14:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:23.017+0000 7fbd63c98700 1 -- 172.21.15.32:0/2961347868 wait complete. 2024-04-23T18:14:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:14:23.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:23 smithi032 ceph-mon[29221]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:23.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:23 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:23 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:23 smithi040 ceph-mon[32626]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:23 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:23 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.657 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-04-23T18:14:22.084704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653326,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":572,"num_read_kb":404,"num_write":339,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181900,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377588,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564236,"statfs":{"total":1151940427776,"available":1151553777664,"internally_reserved":0,"allocated":11993088,"data_stored":7480146,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195614+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:13:57.195614+0000","last_peered":"2024-04-23T18:13:57.195614+0000","last_clean":"2024-04-23T18:13:57.195614+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:13:57.195614+0000","last_undegraded":"2024-04-23T18:13:57.195614+0000","last_fullsized":"2024-04-23T18:13:57.195614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201308+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:13:57.201308+0000","last_peered":"2024-04-23T18:13:57.201308+0000","last_clean":"2024-04-23T18:13:57.201308+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:13:57.201308+0000","last_undegraded":"2024-04-23T18:13:57.201308+0000","last_fullsized":"2024-04-23T18:13:57.201308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201217+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:13:57.201217+0000","last_peered":"2024-04-23T18:13:57.201217+0000","last_clean":"2024-04-23T18:13:57.201217+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:13:57.201217+0000","last_undegraded":"2024-04-23T18:13:57.201217+0000","last_fullsized":"2024-04-23T18:13:57.201217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291408+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:13:57.291408+0000","last_peered":"2024-04-23T18:13:57.291408+0000","last_clean":"2024-04-23T18:13:57.291408+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:13:57.291408+0000","last_undegraded":"2024-04-23T18:13:57.291408+0000","last_fullsized":"2024-04-23T18:13:57.291408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197306+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:13:57.197306+0000","last_peered":"2024-04-23T18:13:57.197306+0000","last_clean":"2024-04-23T18:13:57.197306+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:13:57.197306+0000","last_undegraded":"2024-04-23T18:13:57.197306+0000","last_fullsized":"2024-04-23T18:13:57.197306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197747+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:13:57.197747+0000","last_peered":"2024-04-23T18:13:57.197747+0000","last_clean":"2024-04-23T18:13:57.197747+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:13:57.197747+0000","last_undegraded":"2024-04-23T18:13:57.197747+0000","last_fullsized":"2024-04-23T18:13:57.197747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196843+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:13:57.196843+0000","last_peered":"2024-04-23T18:13:57.196843+0000","last_clean":"2024-04-23T18:13:57.196843+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:13:57.196843+0000","last_undegraded":"2024-04-23T18:13:57.196843+0000","last_fullsized":"2024-04-23T18:13:57.196843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199413+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:13:57.199413+0000","last_peered":"2024-04-23T18:13:57.199413+0000","last_clean":"2024-04-23T18:13:57.199413+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:13:57.199413+0000","last_undegraded":"2024-04-23T18:13:57.199413+0000","last_fullsized":"2024-04-23T18:13:57.199413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199463+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:13:57.199463+0000","last_peered":"2024-04-23T18:13:57.199463+0000","last_clean":"2024-04-23T18:13:57.199463+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:13:57.199463+0000","last_undegraded":"2024-04-23T18:13:57.199463+0000","last_fullsized":"2024-04-23T18:13:57.199463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198238+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:13:57.198238+0000","last_peered":"2024-04-23T18:13:57.198238+0000","last_clean":"2024-04-23T18:13:57.198238+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:13:57.198238+0000","last_undegraded":"2024-04-23T18:13:57.198238+0000","last_fullsized":"2024-04-23T18:13:57.198238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197448+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:13:57.197448+0000","last_peered":"2024-04-23T18:13:57.197448+0000","last_clean":"2024-04-23T18:13:57.197448+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:13:57.197448+0000","last_undegraded":"2024-04-23T18:13:57.197448+0000","last_fullsized":"2024-04-23T18:13:57.197448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291361+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:13:57.291361+0000","last_peered":"2024-04-23T18:13:57.291361+0000","last_clean":"2024-04-23T18:13:57.291361+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:13:57.291361+0000","last_undegraded":"2024-04-23T18:13:57.291361+0000","last_fullsized":"2024-04-23T18:13:57.291361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290631+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:13:57.290631+0000","last_peered":"2024-04-23T18:13:57.290631+0000","last_clean":"2024-04-23T18:13:57.290631+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:13:57.290631+0000","last_undegraded":"2024-04-23T18:13:57.290631+0000","last_fullsized":"2024-04-23T18:13:57.290631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201150+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:13:57.201150+0000","last_peered":"2024-04-23T18:13:57.201150+0000","last_clean":"2024-04-23T18:13:57.201150+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:13:57.201150+0000","last_undegraded":"2024-04-23T18:13:57.201150+0000","last_fullsized":"2024-04-23T18:13:57.201150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.200978+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:13:57.200978+0000","last_peered":"2024-04-23T18:13:57.200978+0000","last_clean":"2024-04-23T18:13:57.200978+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:13:57.200978+0000","last_undegraded":"2024-04-23T18:13:57.200978+0000","last_fullsized":"2024-04-23T18:13:57.200978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198343+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:13:57.198343+0000","last_peered":"2024-04-23T18:13:57.198343+0000","last_clean":"2024-04-23T18:13:57.198343+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:13:57.198343+0000","last_undegraded":"2024-04-23T18:13:57.198343+0000","last_fullsized":"2024-04-23T18:13:57.198343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195883+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:13:57.195883+0000","last_peered":"2024-04-23T18:13:57.195883+0000","last_clean":"2024-04-23T18:13:57.195883+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:13:57.195883+0000","last_undegraded":"2024-04-23T18:13:57.195883+0000","last_fullsized":"2024-04-23T18:13:57.195883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.194091+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:13:57.194091+0000","last_peered":"2024-04-23T18:13:57.194091+0000","last_clean":"2024-04-23T18:13:57.194091+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:13:57.194091+0000","last_undegraded":"2024-04-23T18:13:57.194091+0000","last_fullsized":"2024-04-23T18:13:57.194091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199399+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:13:57.199399+0000","last_peered":"2024-04-23T18:13:57.199399+0000","last_clean":"2024-04-23T18:13:57.199399+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:13:57.199399+0000","last_undegraded":"2024-04-23T18:13:57.199399+0000","last_fullsized":"2024-04-23T18:13:57.199399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.299210+0000","last_change":"2024-04-23T18:13:57.299210+0000","last_active":"2024-04-23T18:13:57.299210+0000","last_peered":"2024-04-23T18:13:57.299210+0000","last_clean":"2024-04-23T18:13:57.299210+0000","last_became_active":"2024-04-23T18:13:57.292381+0000","last_became_peered":"2024-04-23T18:13:57.292381+0000","last_unstale":"2024-04-23T18:13:57.299210+0000","last_undegraded":"2024-04-23T18:13:57.299210+0000","last_fullsized":"2024-04-23T18:13:57.299210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:42.677179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199760+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:13:57.199760+0000","last_peered":"2024-04-23T18:13:57.199760+0000","last_clean":"2024-04-23T18:13:57.199760+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:13:57.199760+0000","last_undegraded":"2024-04-23T18:13:57.199760+0000","last_fullsized":"2024-04-23T18:13:57.199760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199975+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:13:57.199975+0000","last_peered":"2024-04-23T18:13:57.199975+0000","last_clean":"2024-04-23T18:13:57.199975+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:13:57.199975+0000","last_undegraded":"2024-04-23T18:13:57.199975+0000","last_fullsized":"2024-04-23T18:13:57.199975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201588+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:13:57.201588+0000","last_peered":"2024-04-23T18:13:57.201588+0000","last_clean":"2024-04-23T18:13:57.201588+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:13:57.201588+0000","last_undegraded":"2024-04-23T18:13:57.201588+0000","last_fullsized":"2024-04-23T18:13:57.201588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.205351+0000","last_change":"2024-04-23T18:13:57.204788+0000","last_active":"2024-04-23T18:13:57.205351+0000","last_peered":"2024-04-23T18:13:57.205351+0000","last_clean":"2024-04-23T18:13:57.205351+0000","last_became_active":"2024-04-23T18:13:57.199938+0000","last_became_peered":"2024-04-23T18:13:57.199938+0000","last_unstale":"2024-04-23T18:13:57.205351+0000","last_undegraded":"2024-04-23T18:13:57.205351+0000","last_fullsized":"2024-04-23T18:13:57.205351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:10.411355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199350+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:13:57.199350+0000","last_peered":"2024-04-23T18:13:57.199350+0000","last_clean":"2024-04-23T18:13:57.199350+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:13:57.199350+0000","last_undegraded":"2024-04-23T18:13:57.199350+0000","last_fullsized":"2024-04-23T18:13:57.199350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197364+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:13:57.197364+0000","last_peered":"2024-04-23T18:13:57.197364+0000","last_clean":"2024-04-23T18:13:57.197364+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:13:57.197364+0000","last_undegraded":"2024-04-23T18:13:57.197364+0000","last_fullsized":"2024-04-23T18:13:57.197364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290956+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:13:57.290956+0000","last_peered":"2024-04-23T18:13:57.290956+0000","last_clean":"2024-04-23T18:13:57.290956+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:13:57.290956+0000","last_undegraded":"2024-04-23T18:13:57.290956+0000","last_fullsized":"2024-04-23T18:13:57.290956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:16.152987+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:14:16.152987+0000","last_peered":"2024-04-23T18:14:16.152987+0000","last_clean":"2024-04-23T18:14:16.152987+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:14:16.152987+0000","last_undegraded":"2024-04-23T18:14:16.152987+0000","last_fullsized":"2024-04-23T18:14:16.152987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196972+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:13:57.196972+0000","last_peered":"2024-04-23T18:13:57.196972+0000","last_clean":"2024-04-23T18:13:57.196972+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:13:57.196972+0000","last_undegraded":"2024-04-23T18:13:57.196972+0000","last_fullsized":"2024-04-23T18:13:57.196972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197615+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:13:57.197615+0000","last_peered":"2024-04-23T18:13:57.197615+0000","last_clean":"2024-04-23T18:13:57.197615+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:13:57.197615+0000","last_undegraded":"2024-04-23T18:13:57.197615+0000","last_fullsized":"2024-04-23T18:13:57.197615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290956+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:13:57.290956+0000","last_peered":"2024-04-23T18:13:57.290956+0000","last_clean":"2024-04-23T18:13:57.290956+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:13:57.290956+0000","last_undegraded":"2024-04-23T18:13:57.290956+0000","last_fullsized":"2024-04-23T18:13:57.290956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290550+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:13:57.290550+0000","last_peered":"2024-04-23T18:13:57.290550+0000","last_clean":"2024-04-23T18:13:57.290550+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:13:57.290550+0000","last_undegraded":"2024-04-23T18:13:57.290550+0000","last_fullsized":"2024-04-23T18:13:57.290550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199216+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:13:57.199216+0000","last_peered":"2024-04-23T18:13:57.199216+0000","last_clean":"2024-04-23T18:13:57.199216+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:13:57.199216+0000","last_undegraded":"2024-04-23T18:13:57.199216+0000","last_fullsized":"2024-04-23T18:13:57.199216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290645+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:13:57.290645+0000","last_peered":"2024-04-23T18:13:57.290645+0000","last_clean":"2024-04-23T18:13:57.290645+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:13:57.290645+0000","last_undegraded":"2024-04-23T18:13:57.290645+0000","last_fullsized":"2024-04-23T18:13:57.290645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.193836+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:13:57.193836+0000","last_peered":"2024-04-23T18:13:57.193836+0000","last_clean":"2024-04-23T18:13:57.193836+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:13:57.193836+0000","last_undegraded":"2024-04-23T18:13:57.193836+0000","last_fullsized":"2024-04-23T18:13:57.193836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:17.153740+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:14:17.153740+0000","last_peered":"2024-04-23T18:14:17.153740+0000","last_clean":"2024-04-23T18:14:17.153740+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:14:17.153740+0000","last_undegraded":"2024-04-23T18:14:17.153740+0000","last_fullsized":"2024-04-23T18:14:17.153740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198929+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:13:57.198929+0000","last_peered":"2024-04-23T18:13:57.198929+0000","last_clean":"2024-04-23T18:13:57.198929+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:13:57.198929+0000","last_undegraded":"2024-04-23T18:13:57.198929+0000","last_fullsized":"2024-04-23T18:13:57.198929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197363+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:13:57.197363+0000","last_peered":"2024-04-23T18:13:57.197363+0000","last_clean":"2024-04-23T18:13:57.197363+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:13:57.197363+0000","last_undegraded":"2024-04-23T18:13:57.197363+0000","last_fullsized":"2024-04-23T18:13:57.197363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201518+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:13:57.201518+0000","last_peered":"2024-04-23T18:13:57.201518+0000","last_clean":"2024-04-23T18:13:57.201518+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:13:57.201518+0000","last_undegraded":"2024-04-23T18:13:57.201518+0000","last_fullsized":"2024-04-23T18:13:57.201518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291038+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:13:57.291038+0000","last_peered":"2024-04-23T18:13:57.291038+0000","last_clean":"2024-04-23T18:13:57.291038+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:13:57.291038+0000","last_undegraded":"2024-04-23T18:13:57.291038+0000","last_fullsized":"2024-04-23T18:13:57.291038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199586+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:57.199586+0000","last_peered":"2024-04-23T18:13:57.199586+0000","last_clean":"2024-04-23T18:13:57.199586+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:13:57.199586+0000","last_undegraded":"2024-04-23T18:13:57.199586+0000","last_fullsized":"2024-04-23T18:13:57.199586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":191,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:18.156735+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:14:18.156735+0000","last_peered":"2024-04-23T18:14:18.156735+0000","last_clean":"2024-04-23T18:14:18.156735+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:14:18.156735+0000","last_undegraded":"2024-04-23T18:14:18.156735+0000","last_fullsized":"2024-04-23T18:14:18.156735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58332,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":129,"num_write":28,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197611+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:13:57.197611+0000","last_peered":"2024-04-23T18:13:57.197611+0000","last_clean":"2024-04-23T18:13:57.197611+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:13:57.197611+0000","last_undegraded":"2024-04-23T18:13:57.197611+0000","last_fullsized":"2024-04-23T18:13:57.197611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198671+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:13:57.198671+0000","last_peered":"2024-04-23T18:13:57.198671+0000","last_clean":"2024-04-23T18:13:57.198671+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:13:57.198671+0000","last_undegraded":"2024-04-23T18:13:57.198671+0000","last_fullsized":"2024-04-23T18:13:57.198671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290556+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:13:57.290556+0000","last_peered":"2024-04-23T18:13:57.290556+0000","last_clean":"2024-04-23T18:13:57.290556+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:13:57.290556+0000","last_undegraded":"2024-04-23T18:13:57.290556+0000","last_fullsized":"2024-04-23T18:13:57.290556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290740+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:13:57.290740+0000","last_peered":"2024-04-23T18:13:57.290740+0000","last_clean":"2024-04-23T18:13:57.290740+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:13:57.290740+0000","last_undegraded":"2024-04-23T18:13:57.290740+0000","last_fullsized":"2024-04-23T18:13:57.290740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199068+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:57.199068+0000","last_peered":"2024-04-23T18:13:57.199068+0000","last_clean":"2024-04-23T18:13:57.199068+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:13:57.199068+0000","last_undegraded":"2024-04-23T18:13:57.199068+0000","last_fullsized":"2024-04-23T18:13:57.199068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:15.153987+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:14:15.153987+0000","last_peered":"2024-04-23T18:14:15.153987+0000","last_clean":"2024-04-23T18:14:15.153987+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:14:15.153987+0000","last_undegraded":"2024-04-23T18:14:15.153987+0000","last_fullsized":"2024-04-23T18:14:15.153987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196391+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:13:57.196391+0000","last_peered":"2024-04-23T18:13:57.196391+0000","last_clean":"2024-04-23T18:13:57.196391+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:13:57.196391+0000","last_undegraded":"2024-04-23T18:13:57.196391+0000","last_fullsized":"2024-04-23T18:13:57.196391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196586+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:13:57.196586+0000","last_peered":"2024-04-23T18:13:57.196586+0000","last_clean":"2024-04-23T18:13:57.196586+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:13:57.196586+0000","last_undegraded":"2024-04-23T18:13:57.196586+0000","last_fullsized":"2024-04-23T18:13:57.196586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290630+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:57.290630+0000","last_peered":"2024-04-23T18:13:57.290630+0000","last_clean":"2024-04-23T18:13:57.290630+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:13:57.290630+0000","last_undegraded":"2024-04-23T18:13:57.290630+0000","last_fullsized":"2024-04-23T18:13:57.290630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197972+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:13:57.197972+0000","last_peered":"2024-04-23T18:13:57.197972+0000","last_clean":"2024-04-23T18:13:57.197972+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:13:57.197972+0000","last_undegraded":"2024-04-23T18:13:57.197972+0000","last_fullsized":"2024-04-23T18:13:57.197972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.292094+0000","last_change":"2024-04-23T18:13:57.292094+0000","last_active":"2024-04-23T18:13:57.292094+0000","last_peered":"2024-04-23T18:13:57.292094+0000","last_clean":"2024-04-23T18:13:57.292094+0000","last_became_active":"2024-04-23T18:13:57.291761+0000","last_became_peered":"2024-04-23T18:13:57.291761+0000","last_unstale":"2024-04-23T18:13:57.292094+0000","last_undegraded":"2024-04-23T18:13:57.292094+0000","last_fullsized":"2024-04-23T18:13:57.292094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:38:31.544598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.193975+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:13:57.193975+0000","last_peered":"2024-04-23T18:13:57.193975+0000","last_clean":"2024-04-23T18:13:57.193975+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:13:57.193975+0000","last_undegraded":"2024-04-23T18:13:57.193975+0000","last_fullsized":"2024-04-23T18:13:57.193975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199340+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:13:57.199340+0000","last_peered":"2024-04-23T18:13:57.199340+0000","last_clean":"2024-04-23T18:13:57.199340+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:13:57.199340+0000","last_undegraded":"2024-04-23T18:13:57.199340+0000","last_fullsized":"2024-04-23T18:13:57.199340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196750+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:13:57.196750+0000","last_peered":"2024-04-23T18:13:57.196750+0000","last_clean":"2024-04-23T18:13:57.196750+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:13:57.196750+0000","last_undegraded":"2024-04-23T18:13:57.196750+0000","last_fullsized":"2024-04-23T18:13:57.196750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.202859+0000","last_change":"2024-04-23T18:13:57.202859+0000","last_active":"2024-04-23T18:13:57.202859+0000","last_peered":"2024-04-23T18:13:57.202859+0000","last_clean":"2024-04-23T18:13:57.202859+0000","last_became_active":"2024-04-23T18:13:57.202395+0000","last_became_peered":"2024-04-23T18:13:57.202395+0000","last_unstale":"2024-04-23T18:13:57.202859+0000","last_undegraded":"2024-04-23T18:13:57.202859+0000","last_fullsized":"2024-04-23T18:13:57.202859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:17.871010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291056+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:13:57.291056+0000","last_peered":"2024-04-23T18:13:57.291056+0000","last_clean":"2024-04-23T18:13:57.291056+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:13:57.291056+0000","last_undegraded":"2024-04-23T18:13:57.291056+0000","last_fullsized":"2024-04-23T18:13:57.291056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:16.154895+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:14:16.154895+0000","last_peered":"2024-04-23T18:14:16.154895+0000","last_clean":"2024-04-23T18:14:16.154895+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:14:16.154895+0000","last_undegraded":"2024-04-23T18:14:16.154895+0000","last_fullsized":"2024-04-23T18:14:16.154895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.200025+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:13:57.200025+0000","last_peered":"2024-04-23T18:13:57.200025+0000","last_clean":"2024-04-23T18:13:57.200025+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:13:57.200025+0000","last_undegraded":"2024-04-23T18:13:57.200025+0000","last_fullsized":"2024-04-23T18:13:57.200025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291418+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:13:57.291418+0000","last_peered":"2024-04-23T18:13:57.291418+0000","last_clean":"2024-04-23T18:13:57.291418+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:13:57.291418+0000","last_undegraded":"2024-04-23T18:13:57.291418+0000","last_fullsized":"2024-04-23T18:13:57.291418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196629+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:13:57.196629+0000","last_peered":"2024-04-23T18:13:57.196629+0000","last_clean":"2024-04-23T18:13:57.196629+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:13:57.196629+0000","last_undegraded":"2024-04-23T18:13:57.196629+0000","last_fullsized":"2024-04-23T18:13:57.196629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290423+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:13:57.290423+0000","last_peered":"2024-04-23T18:13:57.290423+0000","last_clean":"2024-04-23T18:13:57.290423+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:13:57.290423+0000","last_undegraded":"2024-04-23T18:13:57.290423+0000","last_fullsized":"2024-04-23T18:13:57.290423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196839+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:13:57.196839+0000","last_peered":"2024-04-23T18:13:57.196839+0000","last_clean":"2024-04-23T18:13:57.196839+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:13:57.196839+0000","last_undegraded":"2024-04-23T18:13:57.196839+0000","last_fullsized":"2024-04-23T18:13:57.196839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197809+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:13:57.197809+0000","last_peered":"2024-04-23T18:13:57.197809+0000","last_clean":"2024-04-23T18:13:57.197809+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:13:57.197809+0000","last_undegraded":"2024-04-23T18:13:57.197809+0000","last_fullsized":"2024-04-23T18:13:57.197809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290756+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:13:57.290756+0000","last_peered":"2024-04-23T18:13:57.290756+0000","last_clean":"2024-04-23T18:13:57.290756+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:13:57.290756+0000","last_undegraded":"2024-04-23T18:13:57.290756+0000","last_fullsized":"2024-04-23T18:13:57.290756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.194072+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:13:57.194072+0000","last_peered":"2024-04-23T18:13:57.194072+0000","last_clean":"2024-04-23T18:13:57.194072+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:13:57.194072+0000","last_undegraded":"2024-04-23T18:13:57.194072+0000","last_fullsized":"2024-04-23T18:13:57.194072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199834+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:13:57.199834+0000","last_peered":"2024-04-23T18:13:57.199834+0000","last_clean":"2024-04-23T18:13:57.199834+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:13:57.199834+0000","last_undegraded":"2024-04-23T18:13:57.199834+0000","last_fullsized":"2024-04-23T18:13:57.199834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291560+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:13:57.291560+0000","last_peered":"2024-04-23T18:13:57.291560+0000","last_clean":"2024-04-23T18:13:57.291560+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:13:57.291560+0000","last_undegraded":"2024-04-23T18:13:57.291560+0000","last_fullsized":"2024-04-23T18:13:57.291560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201371+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:13:57.201371+0000","last_peered":"2024-04-23T18:13:57.201371+0000","last_clean":"2024-04-23T18:13:57.201371+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:13:57.201371+0000","last_undegraded":"2024-04-23T18:13:57.201371+0000","last_fullsized":"2024-04-23T18:13:57.201371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198476+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:13:57.198476+0000","last_peered":"2024-04-23T18:13:57.198476+0000","last_clean":"2024-04-23T18:13:57.198476+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:13:57.198476+0000","last_undegraded":"2024-04-23T18:13:57.198476+0000","last_fullsized":"2024-04-23T18:13:57.198476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196746+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:13:57.196746+0000","last_peered":"2024-04-23T18:13:57.196746+0000","last_clean":"2024-04-23T18:13:57.196746+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:13:57.196746+0000","last_undegraded":"2024-04-23T18:13:57.196746+0000","last_fullsized":"2024-04-23T18:13:57.196746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199662+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:13:57.199662+0000","last_peered":"2024-04-23T18:13:57.199662+0000","last_clean":"2024-04-23T18:13:57.199662+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:13:57.199662+0000","last_undegraded":"2024-04-23T18:13:57.199662+0000","last_fullsized":"2024-04-23T18:13:57.199662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195262+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:13:57.195262+0000","last_peered":"2024-04-23T18:13:57.195262+0000","last_clean":"2024-04-23T18:13:57.195262+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:13:57.195262+0000","last_undegraded":"2024-04-23T18:13:57.195262+0000","last_fullsized":"2024-04-23T18:13:57.195262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.208103+0000","last_change":"2024-04-23T18:13:57.207521+0000","last_active":"2024-04-23T18:13:57.208103+0000","last_peered":"2024-04-23T18:13:57.208103+0000","last_clean":"2024-04-23T18:13:57.208103+0000","last_became_active":"2024-04-23T18:13:57.201507+0000","last_became_peered":"2024-04-23T18:13:57.201507+0000","last_unstale":"2024-04-23T18:13:57.208103+0000","last_undegraded":"2024-04-23T18:13:57.208103+0000","last_fullsized":"2024-04-23T18:13:57.208103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:35.254723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199699+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:13:57.199699+0000","last_peered":"2024-04-23T18:13:57.199699+0000","last_clean":"2024-04-23T18:13:57.199699+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:13:57.199699+0000","last_undegraded":"2024-04-23T18:13:57.199699+0000","last_fullsized":"2024-04-23T18:13:57.199699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197518+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:13:57.197518+0000","last_peered":"2024-04-23T18:13:57.197518+0000","last_clean":"2024-04-23T18:13:57.197518+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:13:57.197518+0000","last_undegraded":"2024-04-23T18:13:57.197518+0000","last_fullsized":"2024-04-23T18:13:57.197518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197939+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:13:57.197939+0000","last_peered":"2024-04-23T18:13:57.197939+0000","last_clean":"2024-04-23T18:13:57.197939+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:13:57.197939+0000","last_undegraded":"2024-04-23T18:13:57.197939+0000","last_fullsized":"2024-04-23T18:13:57.197939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.206711+0000","last_change":"2024-04-23T18:13:57.206276+0000","last_active":"2024-04-23T18:13:57.206711+0000","last_peered":"2024-04-23T18:13:57.206711+0000","last_clean":"2024-04-23T18:13:57.206711+0000","last_became_active":"2024-04-23T18:13:57.202137+0000","last_became_peered":"2024-04-23T18:13:57.202137+0000","last_unstale":"2024-04-23T18:13:57.206711+0000","last_undegraded":"2024-04-23T18:13:57.206711+0000","last_fullsized":"2024-04-23T18:13:57.206711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:12.998331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197525+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:13:57.197525+0000","last_peered":"2024-04-23T18:13:57.197525+0000","last_clean":"2024-04-23T18:13:57.197525+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:13:57.197525+0000","last_undegraded":"2024-04-23T18:13:57.197525+0000","last_fullsized":"2024-04-23T18:13:57.197525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290821+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:13:57.290821+0000","last_peered":"2024-04-23T18:13:57.290821+0000","last_clean":"2024-04-23T18:13:57.290821+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:13:57.290821+0000","last_undegraded":"2024-04-23T18:13:57.290821+0000","last_fullsized":"2024-04-23T18:13:57.290821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197178+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:13:57.197178+0000","last_peered":"2024-04-23T18:13:57.197178+0000","last_clean":"2024-04-23T18:13:57.197178+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:13:57.197178+0000","last_undegraded":"2024-04-23T18:13:57.197178+0000","last_fullsized":"2024-04-23T18:13:57.197178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290694+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:13:57.290694+0000","last_peered":"2024-04-23T18:13:57.290694+0000","last_clean":"2024-04-23T18:13:57.290694+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:13:57.290694+0000","last_undegraded":"2024-04-23T18:13:57.290694+0000","last_fullsized":"2024-04-23T18:13:57.290694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197682+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:13:57.197682+0000","last_peered":"2024-04-23T18:13:57.197682+0000","last_clean":"2024-04-23T18:13:57.197682+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:13:57.197682+0000","last_undegraded":"2024-04-23T18:13:57.197682+0000","last_fullsized":"2024-04-23T18:13:57.197682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195374+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:13:57.195374+0000","last_peered":"2024-04-23T18:13:57.195374+0000","last_clean":"2024-04-23T18:13:57.195374+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:13:57.195374+0000","last_undegraded":"2024-04-23T18:13:57.195374+0000","last_fullsized":"2024-04-23T18:13:57.195374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198199+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:13:57.198199+0000","last_peered":"2024-04-23T18:13:57.198199+0000","last_clean":"2024-04-23T18:13:57.198199+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:13:57.198199+0000","last_undegraded":"2024-04-23T18:13:57.198199+0000","last_fullsized":"2024-04-23T18:13:57.198199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198357+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:13:57.198357+0000","last_peered":"2024-04-23T18:13:57.198357+0000","last_clean":"2024-04-23T18:13:57.198357+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:13:57.198357+0000","last_undegraded":"2024-04-23T18:13:57.198357+0000","last_fullsized":"2024-04-23T18:13:57.198357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201366+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:13:57.201366+0000","last_peered":"2024-04-23T18:13:57.201366+0000","last_clean":"2024-04-23T18:13:57.201366+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:13:57.201366+0000","last_undegraded":"2024-04-23T18:13:57.201366+0000","last_fullsized":"2024-04-23T18:13:57.201366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290657+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:13:57.290657+0000","last_peered":"2024-04-23T18:13:57.290657+0000","last_clean":"2024-04-23T18:13:57.290657+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:13:57.290657+0000","last_undegraded":"2024-04-23T18:13:57.290657+0000","last_fullsized":"2024-04-23T18:13:57.290657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196071+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:13:57.196071+0000","last_peered":"2024-04-23T18:13:57.196071+0000","last_clean":"2024-04-23T18:13:57.196071+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:13:57.196071+0000","last_undegraded":"2024-04-23T18:13:57.196071+0000","last_fullsized":"2024-04-23T18:13:57.196071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199684+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:13:57.199684+0000","last_peered":"2024-04-23T18:13:57.199684+0000","last_clean":"2024-04-23T18:13:57.199684+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:13:57.199684+0000","last_undegraded":"2024-04-23T18:13:57.199684+0000","last_fullsized":"2024-04-23T18:13:57.199684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199804+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:13:57.199804+0000","last_peered":"2024-04-23T18:13:57.199804+0000","last_clean":"2024-04-23T18:13:57.199804+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:13:57.199804+0000","last_undegraded":"2024-04-23T18:13:57.199804+0000","last_fullsized":"2024-04-23T18:13:57.199804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290795+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:13:57.290795+0000","last_peered":"2024-04-23T18:13:57.290795+0000","last_clean":"2024-04-23T18:13:57.290795+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:13:57.290795+0000","last_undegraded":"2024-04-23T18:13:57.290795+0000","last_fullsized":"2024-04-23T18:13:57.290795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199550+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:13:57.199550+0000","last_peered":"2024-04-23T18:13:57.199550+0000","last_clean":"2024-04-23T18:13:57.199550+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:13:57.199550+0000","last_undegraded":"2024-04-23T18:13:57.199550+0000","last_fullsized":"2024-04-23T18:13:57.199550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198643+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:13:57.198643+0000","last_peered":"2024-04-23T18:13:57.198643+0000","last_clean":"2024-04-23T18:13:57.198643+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:13:57.198643+0000","last_undegraded":"2024-04-23T18:13:57.198643+0000","last_fullsized":"2024-04-23T18:13:57.198643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290681+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:13:57.290681+0000","last_peered":"2024-04-23T18:13:57.290681+0000","last_clean":"2024-04-23T18:13:57.290681+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:13:57.290681+0000","last_undegraded":"2024-04-23T18:13:57.290681+0000","last_fullsized":"2024-04-23T18:13:57.290681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199427+0000","last_change":"2024-04-23T18:13:57.199427+0000","last_active":"2024-04-23T18:13:57.199427+0000","last_peered":"2024-04-23T18:13:57.199427+0000","last_clean":"2024-04-23T18:13:57.199427+0000","last_became_active":"2024-04-23T18:13:57.199001+0000","last_became_peered":"2024-04-23T18:13:57.199001+0000","last_unstale":"2024-04-23T18:13:57.199427+0000","last_undegraded":"2024-04-23T18:13:57.199427+0000","last_fullsized":"2024-04-23T18:13:57.199427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:14.788961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249130,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62724,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":436,"num_read_kb":285,"num_write":156,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1164,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":460596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":1110800,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":460170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":1053073,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":460221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":461492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":519505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1052366,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":58914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":59491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:14:23.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:14:23.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:23 smithi064 ceph-mon[33062]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:23.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:23 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:23 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:24.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:24 smithi032 ceph-mon[29221]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:24.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:24.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:24.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:24 smithi032 ceph-mon[29221]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:24 smithi040 ceph-mon[32626]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:24.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:24 smithi040 ceph-mon[32626]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:24.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:24 smithi064 ceph-mon[33062]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:24.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:24.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:24.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:24 smithi064 ceph-mon[33062]: from='mgr.14152 172.21.15.32:0/2834357868' entity='mgr.x' 2024-04-23T18:14:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 -- 172.21.15.32:0/2138521286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c06b1f0 msgr2=0x7f136c06b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 --2- 172.21.15.32:0/2138521286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c06b1f0 0x7f136c06b670 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f135c009990 tx=0x7f135c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 -- 172.21.15.32:0/2138521286 shutdown_connections 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 --2- 172.21.15.32:0/2138521286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 0x7f136c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 --2- 172.21.15.32:0/2138521286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c108a60 0x7f136c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 --2- 172.21.15.32:0/2138521286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c06b1f0 0x7f136c06b670 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 -- 172.21.15.32:0/2138521286 >> 172.21.15.32:0/2138521286 conn(0x7f136c0789e0 msgr2=0x7f136c078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 -- 172.21.15.32:0/2138521286 shutdown_connections 2024-04-23T18:14:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.719+0000 7f13710bd700 1 -- 172.21.15.32:0/2138521286 wait complete. 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.720+0000 7f13710bd700 1 Processor -- start 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.720+0000 7f13710bd700 1 -- start start 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c06b1f0 0x7f136c19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 0x7f136c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f136c112ca0 con 0x7f136c108a60 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f136c112b20 con 0x7f136c06b1f0 2024-04-23T18:14:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f13710bd700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f136c112e20 con 0x7f136c06bbb0 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33484/0 (socket says 172.21.15.32:33484) 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 -- 172.21.15.32:0/2136503506 learned_addr learned my addr 172.21.15.32:0/2136503506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136a59c700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 0x7f136c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136ad9d700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c06b1f0 0x7f136c19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 -- 172.21.15.32:0/2136503506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 msgr2=0x7f136c19b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 0x7f136c19b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 -- 172.21.15.32:0/2136503506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c06b1f0 msgr2=0x7f136c19b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:25.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c06b1f0 0x7f136c19b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 -- 172.21.15.32:0/2136503506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c1a0340 con 0x7f136c108a60 2024-04-23T18:14:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.721+0000 7f136b59e700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f136400b970 tx=0x7f136400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.722+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136400d860 con 0x7f136c108a60 2024-04-23T18:14:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.722+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c1a0630 con 0x7f136c108a60 2024-04-23T18:14:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.722+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1a7eb0 con 0x7f136c108a60 2024-04-23T18:14:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.722+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f136400e430 con 0x7f136c108a60 2024-04-23T18:14:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.722+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364017660 con 0x7f136c108a60 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.723+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c005330 con 0x7f136c108a60 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.724+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f136400da00 con 0x7f136c108a60 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.725+0000 7f1363fff700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 0x7f13580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.725+0000 7f136ad9d700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 0x7f13580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.725+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f13640d0410 con 0x7f136c108a60 2024-04-23T18:14:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.725+0000 7f136ad9d700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 0x7f13580a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f135400ac50 tx=0x7f135400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:25.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.729+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1364013030 con 0x7f136c108a60 2024-04-23T18:14:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.933+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f134c000bc0 con 0x7f13580a3cd0 2024-04-23T18:14:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.940+0000 7f1363fff700 1 -- 172.21.15.32:0/2136503506 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258867 (secure 0 0 0) 0x7f134c000bc0 con 0x7f13580a3cd0 2024-04-23T18:14:25.938 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 msgr2=0x7f13580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 0x7f13580a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f135400ac50 tx=0x7f135400a380 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 msgr2=0x7f136c19fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f136400b970 tx=0x7f136400bc80 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 shutdown_connections 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108a60 0x7f136c19fc00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f13580a3cd0 0x7f13580a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136c06b1f0 0x7f136c19b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 --2- 172.21.15.32:0/2136503506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f136c06bbb0 0x7f136c19b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 >> 172.21.15.32:0/2136503506 conn(0x7f136c0789e0 msgr2=0x7f136c10df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.943+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 shutdown_connections 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:25.944+0000 7f13710bd700 1 -- 172.21.15.32:0/2136503506 wait complete. 2024-04-23T18:14:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:14:26.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:25 smithi032 ceph-mon[29221]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:26.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:25 smithi040 ceph-mon[32626]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:26.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:25 smithi064 ceph-mon[33062]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:26.627 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-04-23T18:14:24.085193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653326,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":602,"num_read_kb":419,"num_write":339,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181900,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377588,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564236,"statfs":{"total":1151940427776,"available":1151553777664,"internally_reserved":0,"allocated":11993088,"data_stored":7480146,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003182"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195614+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:13:57.195614+0000","last_peered":"2024-04-23T18:13:57.195614+0000","last_clean":"2024-04-23T18:13:57.195614+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:13:57.195614+0000","last_undegraded":"2024-04-23T18:13:57.195614+0000","last_fullsized":"2024-04-23T18:13:57.195614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201308+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:13:57.201308+0000","last_peered":"2024-04-23T18:13:57.201308+0000","last_clean":"2024-04-23T18:13:57.201308+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:13:57.201308+0000","last_undegraded":"2024-04-23T18:13:57.201308+0000","last_fullsized":"2024-04-23T18:13:57.201308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201217+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:13:57.201217+0000","last_peered":"2024-04-23T18:13:57.201217+0000","last_clean":"2024-04-23T18:13:57.201217+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:13:57.201217+0000","last_undegraded":"2024-04-23T18:13:57.201217+0000","last_fullsized":"2024-04-23T18:13:57.201217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291408+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:13:57.291408+0000","last_peered":"2024-04-23T18:13:57.291408+0000","last_clean":"2024-04-23T18:13:57.291408+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:13:57.291408+0000","last_undegraded":"2024-04-23T18:13:57.291408+0000","last_fullsized":"2024-04-23T18:13:57.291408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197306+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:13:57.197306+0000","last_peered":"2024-04-23T18:13:57.197306+0000","last_clean":"2024-04-23T18:13:57.197306+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:13:57.197306+0000","last_undegraded":"2024-04-23T18:13:57.197306+0000","last_fullsized":"2024-04-23T18:13:57.197306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197747+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:13:57.197747+0000","last_peered":"2024-04-23T18:13:57.197747+0000","last_clean":"2024-04-23T18:13:57.197747+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:13:57.197747+0000","last_undegraded":"2024-04-23T18:13:57.197747+0000","last_fullsized":"2024-04-23T18:13:57.197747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196843+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:13:57.196843+0000","last_peered":"2024-04-23T18:13:57.196843+0000","last_clean":"2024-04-23T18:13:57.196843+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:13:57.196843+0000","last_undegraded":"2024-04-23T18:13:57.196843+0000","last_fullsized":"2024-04-23T18:13:57.196843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199413+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:13:57.199413+0000","last_peered":"2024-04-23T18:13:57.199413+0000","last_clean":"2024-04-23T18:13:57.199413+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:13:57.199413+0000","last_undegraded":"2024-04-23T18:13:57.199413+0000","last_fullsized":"2024-04-23T18:13:57.199413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199463+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:13:57.199463+0000","last_peered":"2024-04-23T18:13:57.199463+0000","last_clean":"2024-04-23T18:13:57.199463+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:13:57.199463+0000","last_undegraded":"2024-04-23T18:13:57.199463+0000","last_fullsized":"2024-04-23T18:13:57.199463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198238+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:13:57.198238+0000","last_peered":"2024-04-23T18:13:57.198238+0000","last_clean":"2024-04-23T18:13:57.198238+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:13:57.198238+0000","last_undegraded":"2024-04-23T18:13:57.198238+0000","last_fullsized":"2024-04-23T18:13:57.198238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197448+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:13:57.197448+0000","last_peered":"2024-04-23T18:13:57.197448+0000","last_clean":"2024-04-23T18:13:57.197448+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:13:57.197448+0000","last_undegraded":"2024-04-23T18:13:57.197448+0000","last_fullsized":"2024-04-23T18:13:57.197448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291361+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:13:57.291361+0000","last_peered":"2024-04-23T18:13:57.291361+0000","last_clean":"2024-04-23T18:13:57.291361+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:13:57.291361+0000","last_undegraded":"2024-04-23T18:13:57.291361+0000","last_fullsized":"2024-04-23T18:13:57.291361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290631+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:13:57.290631+0000","last_peered":"2024-04-23T18:13:57.290631+0000","last_clean":"2024-04-23T18:13:57.290631+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:13:57.290631+0000","last_undegraded":"2024-04-23T18:13:57.290631+0000","last_fullsized":"2024-04-23T18:13:57.290631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201150+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:13:57.201150+0000","last_peered":"2024-04-23T18:13:57.201150+0000","last_clean":"2024-04-23T18:13:57.201150+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:13:57.201150+0000","last_undegraded":"2024-04-23T18:13:57.201150+0000","last_fullsized":"2024-04-23T18:13:57.201150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.200978+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:13:57.200978+0000","last_peered":"2024-04-23T18:13:57.200978+0000","last_clean":"2024-04-23T18:13:57.200978+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:13:57.200978+0000","last_undegraded":"2024-04-23T18:13:57.200978+0000","last_fullsized":"2024-04-23T18:13:57.200978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198343+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:13:57.198343+0000","last_peered":"2024-04-23T18:13:57.198343+0000","last_clean":"2024-04-23T18:13:57.198343+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:13:57.198343+0000","last_undegraded":"2024-04-23T18:13:57.198343+0000","last_fullsized":"2024-04-23T18:13:57.198343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195883+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:13:57.195883+0000","last_peered":"2024-04-23T18:13:57.195883+0000","last_clean":"2024-04-23T18:13:57.195883+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:13:57.195883+0000","last_undegraded":"2024-04-23T18:13:57.195883+0000","last_fullsized":"2024-04-23T18:13:57.195883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.194091+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:13:57.194091+0000","last_peered":"2024-04-23T18:13:57.194091+0000","last_clean":"2024-04-23T18:13:57.194091+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:13:57.194091+0000","last_undegraded":"2024-04-23T18:13:57.194091+0000","last_fullsized":"2024-04-23T18:13:57.194091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199399+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:13:57.199399+0000","last_peered":"2024-04-23T18:13:57.199399+0000","last_clean":"2024-04-23T18:13:57.199399+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:13:57.199399+0000","last_undegraded":"2024-04-23T18:13:57.199399+0000","last_fullsized":"2024-04-23T18:13:57.199399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.299210+0000","last_change":"2024-04-23T18:13:57.299210+0000","last_active":"2024-04-23T18:13:57.299210+0000","last_peered":"2024-04-23T18:13:57.299210+0000","last_clean":"2024-04-23T18:13:57.299210+0000","last_became_active":"2024-04-23T18:13:57.292381+0000","last_became_peered":"2024-04-23T18:13:57.292381+0000","last_unstale":"2024-04-23T18:13:57.299210+0000","last_undegraded":"2024-04-23T18:13:57.299210+0000","last_fullsized":"2024-04-23T18:13:57.299210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:42.677179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199760+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:13:57.199760+0000","last_peered":"2024-04-23T18:13:57.199760+0000","last_clean":"2024-04-23T18:13:57.199760+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:13:57.199760+0000","last_undegraded":"2024-04-23T18:13:57.199760+0000","last_fullsized":"2024-04-23T18:13:57.199760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199975+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:13:57.199975+0000","last_peered":"2024-04-23T18:13:57.199975+0000","last_clean":"2024-04-23T18:13:57.199975+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:13:57.199975+0000","last_undegraded":"2024-04-23T18:13:57.199975+0000","last_fullsized":"2024-04-23T18:13:57.199975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201588+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:13:57.201588+0000","last_peered":"2024-04-23T18:13:57.201588+0000","last_clean":"2024-04-23T18:13:57.201588+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:13:57.201588+0000","last_undegraded":"2024-04-23T18:13:57.201588+0000","last_fullsized":"2024-04-23T18:13:57.201588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.205351+0000","last_change":"2024-04-23T18:13:57.204788+0000","last_active":"2024-04-23T18:13:57.205351+0000","last_peered":"2024-04-23T18:13:57.205351+0000","last_clean":"2024-04-23T18:13:57.205351+0000","last_became_active":"2024-04-23T18:13:57.199938+0000","last_became_peered":"2024-04-23T18:13:57.199938+0000","last_unstale":"2024-04-23T18:13:57.205351+0000","last_undegraded":"2024-04-23T18:13:57.205351+0000","last_fullsized":"2024-04-23T18:13:57.205351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:10.411355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199350+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:13:57.199350+0000","last_peered":"2024-04-23T18:13:57.199350+0000","last_clean":"2024-04-23T18:13:57.199350+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:13:57.199350+0000","last_undegraded":"2024-04-23T18:13:57.199350+0000","last_fullsized":"2024-04-23T18:13:57.199350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197364+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:13:57.197364+0000","last_peered":"2024-04-23T18:13:57.197364+0000","last_clean":"2024-04-23T18:13:57.197364+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:13:57.197364+0000","last_undegraded":"2024-04-23T18:13:57.197364+0000","last_fullsized":"2024-04-23T18:13:57.197364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290956+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:13:57.290956+0000","last_peered":"2024-04-23T18:13:57.290956+0000","last_clean":"2024-04-23T18:13:57.290956+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:13:57.290956+0000","last_undegraded":"2024-04-23T18:13:57.290956+0000","last_fullsized":"2024-04-23T18:13:57.290956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:21.160633+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:14:21.160633+0000","last_peered":"2024-04-23T18:14:21.160633+0000","last_clean":"2024-04-23T18:14:21.160633+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:14:21.160633+0000","last_undegraded":"2024-04-23T18:14:21.160633+0000","last_fullsized":"2024-04-23T18:14:21.160633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196972+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:13:57.196972+0000","last_peered":"2024-04-23T18:13:57.196972+0000","last_clean":"2024-04-23T18:13:57.196972+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:13:57.196972+0000","last_undegraded":"2024-04-23T18:13:57.196972+0000","last_fullsized":"2024-04-23T18:13:57.196972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197615+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:13:57.197615+0000","last_peered":"2024-04-23T18:13:57.197615+0000","last_clean":"2024-04-23T18:13:57.197615+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:13:57.197615+0000","last_undegraded":"2024-04-23T18:13:57.197615+0000","last_fullsized":"2024-04-23T18:13:57.197615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290956+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:13:57.290956+0000","last_peered":"2024-04-23T18:13:57.290956+0000","last_clean":"2024-04-23T18:13:57.290956+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:13:57.290956+0000","last_undegraded":"2024-04-23T18:13:57.290956+0000","last_fullsized":"2024-04-23T18:13:57.290956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290550+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:13:57.290550+0000","last_peered":"2024-04-23T18:13:57.290550+0000","last_clean":"2024-04-23T18:13:57.290550+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:13:57.290550+0000","last_undegraded":"2024-04-23T18:13:57.290550+0000","last_fullsized":"2024-04-23T18:13:57.290550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199216+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:13:57.199216+0000","last_peered":"2024-04-23T18:13:57.199216+0000","last_clean":"2024-04-23T18:13:57.199216+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:13:57.199216+0000","last_undegraded":"2024-04-23T18:13:57.199216+0000","last_fullsized":"2024-04-23T18:13:57.199216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290645+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:13:57.290645+0000","last_peered":"2024-04-23T18:13:57.290645+0000","last_clean":"2024-04-23T18:13:57.290645+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:13:57.290645+0000","last_undegraded":"2024-04-23T18:13:57.290645+0000","last_fullsized":"2024-04-23T18:13:57.290645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.193836+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:13:57.193836+0000","last_peered":"2024-04-23T18:13:57.193836+0000","last_clean":"2024-04-23T18:13:57.193836+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:13:57.193836+0000","last_undegraded":"2024-04-23T18:13:57.193836+0000","last_fullsized":"2024-04-23T18:13:57.193836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:22.161536+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:14:22.161536+0000","last_peered":"2024-04-23T18:14:22.161536+0000","last_clean":"2024-04-23T18:14:22.161536+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:14:22.161536+0000","last_undegraded":"2024-04-23T18:14:22.161536+0000","last_fullsized":"2024-04-23T18:14:22.161536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198929+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:13:57.198929+0000","last_peered":"2024-04-23T18:13:57.198929+0000","last_clean":"2024-04-23T18:13:57.198929+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:13:57.198929+0000","last_undegraded":"2024-04-23T18:13:57.198929+0000","last_fullsized":"2024-04-23T18:13:57.198929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197363+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:13:57.197363+0000","last_peered":"2024-04-23T18:13:57.197363+0000","last_clean":"2024-04-23T18:13:57.197363+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:13:57.197363+0000","last_undegraded":"2024-04-23T18:13:57.197363+0000","last_fullsized":"2024-04-23T18:13:57.197363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201518+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:13:57.201518+0000","last_peered":"2024-04-23T18:13:57.201518+0000","last_clean":"2024-04-23T18:13:57.201518+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:13:57.201518+0000","last_undegraded":"2024-04-23T18:13:57.201518+0000","last_fullsized":"2024-04-23T18:13:57.201518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291038+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:13:57.291038+0000","last_peered":"2024-04-23T18:13:57.291038+0000","last_clean":"2024-04-23T18:13:57.291038+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:13:57.291038+0000","last_undegraded":"2024-04-23T18:13:57.291038+0000","last_fullsized":"2024-04-23T18:13:57.291038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199586+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:57.199586+0000","last_peered":"2024-04-23T18:13:57.199586+0000","last_clean":"2024-04-23T18:13:57.199586+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:13:57.199586+0000","last_undegraded":"2024-04-23T18:13:57.199586+0000","last_fullsized":"2024-04-23T18:13:57.199586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":201,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:23.163135+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:14:23.163135+0000","last_peered":"2024-04-23T18:14:23.163135+0000","last_clean":"2024-04-23T18:14:23.163135+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:14:23.163135+0000","last_undegraded":"2024-04-23T18:14:23.163135+0000","last_fullsized":"2024-04-23T18:14:23.163135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58332,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":155,"num_read_kb":134,"num_write":28,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197611+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:13:57.197611+0000","last_peered":"2024-04-23T18:13:57.197611+0000","last_clean":"2024-04-23T18:13:57.197611+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:13:57.197611+0000","last_undegraded":"2024-04-23T18:13:57.197611+0000","last_fullsized":"2024-04-23T18:13:57.197611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198671+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:13:57.198671+0000","last_peered":"2024-04-23T18:13:57.198671+0000","last_clean":"2024-04-23T18:13:57.198671+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:13:57.198671+0000","last_undegraded":"2024-04-23T18:13:57.198671+0000","last_fullsized":"2024-04-23T18:13:57.198671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290556+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:13:57.290556+0000","last_peered":"2024-04-23T18:13:57.290556+0000","last_clean":"2024-04-23T18:13:57.290556+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:13:57.290556+0000","last_undegraded":"2024-04-23T18:13:57.290556+0000","last_fullsized":"2024-04-23T18:13:57.290556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290740+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:13:57.290740+0000","last_peered":"2024-04-23T18:13:57.290740+0000","last_clean":"2024-04-23T18:13:57.290740+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:13:57.290740+0000","last_undegraded":"2024-04-23T18:13:57.290740+0000","last_fullsized":"2024-04-23T18:13:57.290740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199068+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:13:57.199068+0000","last_peered":"2024-04-23T18:13:57.199068+0000","last_clean":"2024-04-23T18:13:57.199068+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:13:57.199068+0000","last_undegraded":"2024-04-23T18:13:57.199068+0000","last_fullsized":"2024-04-23T18:13:57.199068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:20.161106+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:14:20.161106+0000","last_peered":"2024-04-23T18:14:20.161106+0000","last_clean":"2024-04-23T18:14:20.161106+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:14:20.161106+0000","last_undegraded":"2024-04-23T18:14:20.161106+0000","last_fullsized":"2024-04-23T18:14:20.161106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196391+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:13:57.196391+0000","last_peered":"2024-04-23T18:13:57.196391+0000","last_clean":"2024-04-23T18:13:57.196391+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:13:57.196391+0000","last_undegraded":"2024-04-23T18:13:57.196391+0000","last_fullsized":"2024-04-23T18:13:57.196391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196586+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:13:57.196586+0000","last_peered":"2024-04-23T18:13:57.196586+0000","last_clean":"2024-04-23T18:13:57.196586+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:13:57.196586+0000","last_undegraded":"2024-04-23T18:13:57.196586+0000","last_fullsized":"2024-04-23T18:13:57.196586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290630+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:13:57.290630+0000","last_peered":"2024-04-23T18:13:57.290630+0000","last_clean":"2024-04-23T18:13:57.290630+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:13:57.290630+0000","last_undegraded":"2024-04-23T18:13:57.290630+0000","last_fullsized":"2024-04-23T18:13:57.290630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197972+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:13:57.197972+0000","last_peered":"2024-04-23T18:13:57.197972+0000","last_clean":"2024-04-23T18:13:57.197972+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:13:57.197972+0000","last_undegraded":"2024-04-23T18:13:57.197972+0000","last_fullsized":"2024-04-23T18:13:57.197972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.292094+0000","last_change":"2024-04-23T18:13:57.292094+0000","last_active":"2024-04-23T18:13:57.292094+0000","last_peered":"2024-04-23T18:13:57.292094+0000","last_clean":"2024-04-23T18:13:57.292094+0000","last_became_active":"2024-04-23T18:13:57.291761+0000","last_became_peered":"2024-04-23T18:13:57.291761+0000","last_unstale":"2024-04-23T18:13:57.292094+0000","last_undegraded":"2024-04-23T18:13:57.292094+0000","last_fullsized":"2024-04-23T18:13:57.292094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:38:31.544598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.193975+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:13:57.193975+0000","last_peered":"2024-04-23T18:13:57.193975+0000","last_clean":"2024-04-23T18:13:57.193975+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:13:57.193975+0000","last_undegraded":"2024-04-23T18:13:57.193975+0000","last_fullsized":"2024-04-23T18:13:57.193975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199340+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:13:57.199340+0000","last_peered":"2024-04-23T18:13:57.199340+0000","last_clean":"2024-04-23T18:13:57.199340+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:13:57.199340+0000","last_undegraded":"2024-04-23T18:13:57.199340+0000","last_fullsized":"2024-04-23T18:13:57.199340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196750+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:13:57.196750+0000","last_peered":"2024-04-23T18:13:57.196750+0000","last_clean":"2024-04-23T18:13:57.196750+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:13:57.196750+0000","last_undegraded":"2024-04-23T18:13:57.196750+0000","last_fullsized":"2024-04-23T18:13:57.196750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.202859+0000","last_change":"2024-04-23T18:13:57.202859+0000","last_active":"2024-04-23T18:13:57.202859+0000","last_peered":"2024-04-23T18:13:57.202859+0000","last_clean":"2024-04-23T18:13:57.202859+0000","last_became_active":"2024-04-23T18:13:57.202395+0000","last_became_peered":"2024-04-23T18:13:57.202395+0000","last_unstale":"2024-04-23T18:13:57.202859+0000","last_undegraded":"2024-04-23T18:13:57.202859+0000","last_fullsized":"2024-04-23T18:13:57.202859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:17.871010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291056+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:13:57.291056+0000","last_peered":"2024-04-23T18:13:57.291056+0000","last_clean":"2024-04-23T18:13:57.291056+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:13:57.291056+0000","last_undegraded":"2024-04-23T18:13:57.291056+0000","last_fullsized":"2024-04-23T18:13:57.291056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:14:21.161853+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:14:21.161853+0000","last_peered":"2024-04-23T18:14:21.161853+0000","last_clean":"2024-04-23T18:14:21.161853+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:14:21.161853+0000","last_undegraded":"2024-04-23T18:14:21.161853+0000","last_fullsized":"2024-04-23T18:14:21.161853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.200025+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:13:57.200025+0000","last_peered":"2024-04-23T18:13:57.200025+0000","last_clean":"2024-04-23T18:13:57.200025+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:13:57.200025+0000","last_undegraded":"2024-04-23T18:13:57.200025+0000","last_fullsized":"2024-04-23T18:13:57.200025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291418+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:13:57.291418+0000","last_peered":"2024-04-23T18:13:57.291418+0000","last_clean":"2024-04-23T18:13:57.291418+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:13:57.291418+0000","last_undegraded":"2024-04-23T18:13:57.291418+0000","last_fullsized":"2024-04-23T18:13:57.291418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196629+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:13:57.196629+0000","last_peered":"2024-04-23T18:13:57.196629+0000","last_clean":"2024-04-23T18:13:57.196629+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:13:57.196629+0000","last_undegraded":"2024-04-23T18:13:57.196629+0000","last_fullsized":"2024-04-23T18:13:57.196629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290423+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:13:57.290423+0000","last_peered":"2024-04-23T18:13:57.290423+0000","last_clean":"2024-04-23T18:13:57.290423+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:13:57.290423+0000","last_undegraded":"2024-04-23T18:13:57.290423+0000","last_fullsized":"2024-04-23T18:13:57.290423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196839+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:13:57.196839+0000","last_peered":"2024-04-23T18:13:57.196839+0000","last_clean":"2024-04-23T18:13:57.196839+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:13:57.196839+0000","last_undegraded":"2024-04-23T18:13:57.196839+0000","last_fullsized":"2024-04-23T18:13:57.196839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197809+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:13:57.197809+0000","last_peered":"2024-04-23T18:13:57.197809+0000","last_clean":"2024-04-23T18:13:57.197809+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:13:57.197809+0000","last_undegraded":"2024-04-23T18:13:57.197809+0000","last_fullsized":"2024-04-23T18:13:57.197809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290756+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:13:57.290756+0000","last_peered":"2024-04-23T18:13:57.290756+0000","last_clean":"2024-04-23T18:13:57.290756+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:13:57.290756+0000","last_undegraded":"2024-04-23T18:13:57.290756+0000","last_fullsized":"2024-04-23T18:13:57.290756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.194072+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:13:57.194072+0000","last_peered":"2024-04-23T18:13:57.194072+0000","last_clean":"2024-04-23T18:13:57.194072+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:13:57.194072+0000","last_undegraded":"2024-04-23T18:13:57.194072+0000","last_fullsized":"2024-04-23T18:13:57.194072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199834+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:13:57.199834+0000","last_peered":"2024-04-23T18:13:57.199834+0000","last_clean":"2024-04-23T18:13:57.199834+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:13:57.199834+0000","last_undegraded":"2024-04-23T18:13:57.199834+0000","last_fullsized":"2024-04-23T18:13:57.199834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.291560+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:13:57.291560+0000","last_peered":"2024-04-23T18:13:57.291560+0000","last_clean":"2024-04-23T18:13:57.291560+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:13:57.291560+0000","last_undegraded":"2024-04-23T18:13:57.291560+0000","last_fullsized":"2024-04-23T18:13:57.291560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201371+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:13:57.201371+0000","last_peered":"2024-04-23T18:13:57.201371+0000","last_clean":"2024-04-23T18:13:57.201371+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:13:57.201371+0000","last_undegraded":"2024-04-23T18:13:57.201371+0000","last_fullsized":"2024-04-23T18:13:57.201371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198476+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:13:57.198476+0000","last_peered":"2024-04-23T18:13:57.198476+0000","last_clean":"2024-04-23T18:13:57.198476+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:13:57.198476+0000","last_undegraded":"2024-04-23T18:13:57.198476+0000","last_fullsized":"2024-04-23T18:13:57.198476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196746+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:13:57.196746+0000","last_peered":"2024-04-23T18:13:57.196746+0000","last_clean":"2024-04-23T18:13:57.196746+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:13:57.196746+0000","last_undegraded":"2024-04-23T18:13:57.196746+0000","last_fullsized":"2024-04-23T18:13:57.196746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199662+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:13:57.199662+0000","last_peered":"2024-04-23T18:13:57.199662+0000","last_clean":"2024-04-23T18:13:57.199662+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:13:57.199662+0000","last_undegraded":"2024-04-23T18:13:57.199662+0000","last_fullsized":"2024-04-23T18:13:57.199662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195262+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:13:57.195262+0000","last_peered":"2024-04-23T18:13:57.195262+0000","last_clean":"2024-04-23T18:13:57.195262+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:13:57.195262+0000","last_undegraded":"2024-04-23T18:13:57.195262+0000","last_fullsized":"2024-04-23T18:13:57.195262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.208103+0000","last_change":"2024-04-23T18:13:57.207521+0000","last_active":"2024-04-23T18:13:57.208103+0000","last_peered":"2024-04-23T18:13:57.208103+0000","last_clean":"2024-04-23T18:13:57.208103+0000","last_became_active":"2024-04-23T18:13:57.201507+0000","last_became_peered":"2024-04-23T18:13:57.201507+0000","last_unstale":"2024-04-23T18:13:57.208103+0000","last_undegraded":"2024-04-23T18:13:57.208103+0000","last_fullsized":"2024-04-23T18:13:57.208103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:35.254723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199699+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:13:57.199699+0000","last_peered":"2024-04-23T18:13:57.199699+0000","last_clean":"2024-04-23T18:13:57.199699+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:13:57.199699+0000","last_undegraded":"2024-04-23T18:13:57.199699+0000","last_fullsized":"2024-04-23T18:13:57.199699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197518+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:13:57.197518+0000","last_peered":"2024-04-23T18:13:57.197518+0000","last_clean":"2024-04-23T18:13:57.197518+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:13:57.197518+0000","last_undegraded":"2024-04-23T18:13:57.197518+0000","last_fullsized":"2024-04-23T18:13:57.197518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197939+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:13:57.197939+0000","last_peered":"2024-04-23T18:13:57.197939+0000","last_clean":"2024-04-23T18:13:57.197939+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:13:57.197939+0000","last_undegraded":"2024-04-23T18:13:57.197939+0000","last_fullsized":"2024-04-23T18:13:57.197939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.206711+0000","last_change":"2024-04-23T18:13:57.206276+0000","last_active":"2024-04-23T18:13:57.206711+0000","last_peered":"2024-04-23T18:13:57.206711+0000","last_clean":"2024-04-23T18:13:57.206711+0000","last_became_active":"2024-04-23T18:13:57.202137+0000","last_became_peered":"2024-04-23T18:13:57.202137+0000","last_unstale":"2024-04-23T18:13:57.206711+0000","last_undegraded":"2024-04-23T18:13:57.206711+0000","last_fullsized":"2024-04-23T18:13:57.206711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:12.998331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197525+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:13:57.197525+0000","last_peered":"2024-04-23T18:13:57.197525+0000","last_clean":"2024-04-23T18:13:57.197525+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:13:57.197525+0000","last_undegraded":"2024-04-23T18:13:57.197525+0000","last_fullsized":"2024-04-23T18:13:57.197525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290821+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:13:57.290821+0000","last_peered":"2024-04-23T18:13:57.290821+0000","last_clean":"2024-04-23T18:13:57.290821+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:13:57.290821+0000","last_undegraded":"2024-04-23T18:13:57.290821+0000","last_fullsized":"2024-04-23T18:13:57.290821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197178+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:13:57.197178+0000","last_peered":"2024-04-23T18:13:57.197178+0000","last_clean":"2024-04-23T18:13:57.197178+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:13:57.197178+0000","last_undegraded":"2024-04-23T18:13:57.197178+0000","last_fullsized":"2024-04-23T18:13:57.197178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290694+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:13:57.290694+0000","last_peered":"2024-04-23T18:13:57.290694+0000","last_clean":"2024-04-23T18:13:57.290694+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:13:57.290694+0000","last_undegraded":"2024-04-23T18:13:57.290694+0000","last_fullsized":"2024-04-23T18:13:57.290694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.197682+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:13:57.197682+0000","last_peered":"2024-04-23T18:13:57.197682+0000","last_clean":"2024-04-23T18:13:57.197682+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:13:57.197682+0000","last_undegraded":"2024-04-23T18:13:57.197682+0000","last_fullsized":"2024-04-23T18:13:57.197682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.195374+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:13:57.195374+0000","last_peered":"2024-04-23T18:13:57.195374+0000","last_clean":"2024-04-23T18:13:57.195374+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:13:57.195374+0000","last_undegraded":"2024-04-23T18:13:57.195374+0000","last_fullsized":"2024-04-23T18:13:57.195374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198199+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:13:57.198199+0000","last_peered":"2024-04-23T18:13:57.198199+0000","last_clean":"2024-04-23T18:13:57.198199+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:13:57.198199+0000","last_undegraded":"2024-04-23T18:13:57.198199+0000","last_fullsized":"2024-04-23T18:13:57.198199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198357+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:13:57.198357+0000","last_peered":"2024-04-23T18:13:57.198357+0000","last_clean":"2024-04-23T18:13:57.198357+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:13:57.198357+0000","last_undegraded":"2024-04-23T18:13:57.198357+0000","last_fullsized":"2024-04-23T18:13:57.198357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.201366+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:13:57.201366+0000","last_peered":"2024-04-23T18:13:57.201366+0000","last_clean":"2024-04-23T18:13:57.201366+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:13:57.201366+0000","last_undegraded":"2024-04-23T18:13:57.201366+0000","last_fullsized":"2024-04-23T18:13:57.201366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290657+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:13:57.290657+0000","last_peered":"2024-04-23T18:13:57.290657+0000","last_clean":"2024-04-23T18:13:57.290657+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:13:57.290657+0000","last_undegraded":"2024-04-23T18:13:57.290657+0000","last_fullsized":"2024-04-23T18:13:57.290657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.196071+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:13:57.196071+0000","last_peered":"2024-04-23T18:13:57.196071+0000","last_clean":"2024-04-23T18:13:57.196071+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:13:57.196071+0000","last_undegraded":"2024-04-23T18:13:57.196071+0000","last_fullsized":"2024-04-23T18:13:57.196071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199684+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:13:57.199684+0000","last_peered":"2024-04-23T18:13:57.199684+0000","last_clean":"2024-04-23T18:13:57.199684+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:13:57.199684+0000","last_undegraded":"2024-04-23T18:13:57.199684+0000","last_fullsized":"2024-04-23T18:13:57.199684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199804+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:13:57.199804+0000","last_peered":"2024-04-23T18:13:57.199804+0000","last_clean":"2024-04-23T18:13:57.199804+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:13:57.199804+0000","last_undegraded":"2024-04-23T18:13:57.199804+0000","last_fullsized":"2024-04-23T18:13:57.199804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290795+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:13:57.290795+0000","last_peered":"2024-04-23T18:13:57.290795+0000","last_clean":"2024-04-23T18:13:57.290795+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:13:57.290795+0000","last_undegraded":"2024-04-23T18:13:57.290795+0000","last_fullsized":"2024-04-23T18:13:57.290795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199550+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:13:57.199550+0000","last_peered":"2024-04-23T18:13:57.199550+0000","last_clean":"2024-04-23T18:13:57.199550+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:13:57.199550+0000","last_undegraded":"2024-04-23T18:13:57.199550+0000","last_fullsized":"2024-04-23T18:13:57.199550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.198643+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:13:57.198643+0000","last_peered":"2024-04-23T18:13:57.198643+0000","last_clean":"2024-04-23T18:13:57.198643+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:13:57.198643+0000","last_undegraded":"2024-04-23T18:13:57.198643+0000","last_fullsized":"2024-04-23T18:13:57.198643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.290681+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:13:57.290681+0000","last_peered":"2024-04-23T18:13:57.290681+0000","last_clean":"2024-04-23T18:13:57.290681+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:13:57.290681+0000","last_undegraded":"2024-04-23T18:13:57.290681+0000","last_fullsized":"2024-04-23T18:13:57.290681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:13:57.199427+0000","last_change":"2024-04-23T18:13:57.199427+0000","last_active":"2024-04-23T18:13:57.199427+0000","last_peered":"2024-04-23T18:13:57.199427+0000","last_clean":"2024-04-23T18:13:57.199427+0000","last_became_active":"2024-04-23T18:13:57.199001+0000","last_became_peered":"2024-04-23T18:13:57.199001+0000","last_unstale":"2024-04-23T18:13:57.199427+0000","last_undegraded":"2024-04-23T18:13:57.199427+0000","last_fullsized":"2024-04-23T18:13:57.199427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:14.788961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249130,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62724,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":466,"num_read_kb":300,"num_write":156,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1164,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":460596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":1110800,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":460170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":1053073,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462968,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":460221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":461492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":519505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920834,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724605,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1052366,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":58914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":59491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:14:26.633 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T18:14:26.634 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T18:14:26.634 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T18:14:26.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T18:14:26.887 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:27.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:26 smithi032 ceph-mon[29221]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:26 smithi064 ceph-mon[33062]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:27.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:26 smithi040 ceph-mon[32626]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:14:27.980 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:27 smithi032 ceph-mon[29221]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:27.980 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:27 smithi032 ceph-mon[29221]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:14:28.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:27 smithi064 ceph-mon[33062]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:28.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:27 smithi064 ceph-mon[33062]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:14:28.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:27 smithi040 ceph-mon[32626]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:28.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:27 smithi040 ceph-mon[32626]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:14:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 -- 172.21.15.32:0/2613776261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 msgr2=0x7f97a810e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 --2- 172.21.15.32:0/2613776261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a810e830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f97a000b0a0 tx=0x7f97a000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 -- 172.21.15.32:0/2613776261 shutdown_connections 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 --2- 172.21.15.32:0/2613776261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a810e830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 --2- 172.21.15.32:0/2613776261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8107960 0x7f97a8107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 --2- 172.21.15.32:0/2613776261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8101960 0x7f97a8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 -- 172.21.15.32:0/2613776261 >> 172.21.15.32:0/2613776261 conn(0x7f97a80789e0 msgr2=0x7f97a8078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 -- 172.21.15.32:0/2613776261 shutdown_connections 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.563+0000 7f97acf50700 1 -- 172.21.15.32:0/2613776261 wait complete. 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 Processor -- start 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 -- start start 2024-04-23T18:14:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8101960 0x7f97a81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8111bf0 con 0x7f97a8102320 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8111a70 con 0x7f97a8107960 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97acf50700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8111d70 con 0x7f97a8101960 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97a6d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97a6d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:34818/0 (socket says 172.21.15.32:34818) 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97a6d9d700 1 -- 172.21.15.32:0/3375454096 learned_addr learned my addr 172.21.15.32:0/3375454096 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97a659c700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8101960 0x7f97a81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.564+0000 7f97a5d9b700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 -- 172.21.15.32:0/3375454096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8101960 msgr2=0x7f97a81980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8101960 0x7f97a81980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 -- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 msgr2=0x7f97a8198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a8198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 -- 172.21.15.32:0/3375454096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a819d0d0 con 0x7f97a8107960 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97a6d9d700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f97a000b070 tx=0x7f97a000ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0021070 con 0x7f97a8107960 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a819d360 con 0x7f97a8107960 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a0007b80 con 0x7f97a8107960 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00052a0 con 0x7f97a8107960 2024-04-23T18:14:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.565+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a81a4c40 con 0x7f97a8107960 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.566+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a8069730 con 0x7f97a8107960 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.568+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f97a0024070 con 0x7f97a8107960 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.568+0000 7f97977fe700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 0x7f97900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.568+0000 7f97a659c700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 0x7f97900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.568+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f97a000f020 con 0x7f97a8107960 2024-04-23T18:14:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.569+0000 7f97a659c700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 0x7f97900a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9798009a10 tx=0x7f9798009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.573+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97a002b020 con 0x7f97a8107960 2024-04-23T18:14:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.826+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f97a8199420 con 0x7f97a8107960 2024-04-23T18:14:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.827+0000 7f97977fe700 1 -- 172.21.15.32:0/3375454096 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f97a00be040 con 0x7f97a8107960 2024-04-23T18:14:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:28.821 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T18:14:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 msgr2=0x7f97900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 0x7f97900a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9798009a10 tx=0x7f9798009190 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 msgr2=0x7f97a819c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f97a000b070 tx=0x7f97a000ce50 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 shutdown_connections 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97900a3cc0 0x7f97900a6180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97a8107960 0x7f97a819c990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97a8101960 0x7f97a81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.829+0000 7f97acf50700 1 --2- 172.21.15.32:0/3375454096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a8102320 0x7f97a8198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.830+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 >> 172.21.15.32:0/3375454096 conn(0x7f97a80789e0 msgr2=0x7f97a81044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.830+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 shutdown_connections 2024-04-23T18:14:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:28.830+0000 7f97acf50700 1 -- 172.21.15.32:0/3375454096 wait complete. 2024-04-23T18:14:29.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:28 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3375454096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:14:29.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:28 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3375454096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:14:29.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:28 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3375454096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:14:29.521 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T18:14:29.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 -- 172.21.15.32:0/2704660257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634102010 msgr2=0x7f0634102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 --2- 172.21.15.32:0/2704660257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634102010 0x7f0634102430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0628009900 tx=0x7f062802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 -- 172.21.15.32:0/2704660257 shutdown_connections 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 --2- 172.21.15.32:0/2704660257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634103bc0 0x7f063410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 --2- 172.21.15.32:0/2704660257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103200 0x7f0634103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 --2- 172.21.15.32:0/2704660257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634102010 0x7f0634102430 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 -- 172.21.15.32:0/2704660257 >> 172.21.15.32:0/2704660257 conn(0x7f06340fd580 msgr2=0x7f06340ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 -- 172.21.15.32:0/2704660257 shutdown_connections 2024-04-23T18:14:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.638+0000 7f063a059640 1 -- 172.21.15.32:0/2704660257 wait complete. 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 Processor -- start 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 -- start start 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 0x7f06341981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103bc0 0x7f063419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f063410dae0 con 0x7f0634103200 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f063410d960 con 0x7f0634102010 2024-04-23T18:14:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f063a059640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f063410dc60 con 0x7f0634103bc0 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f0632ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f0632ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35320/0 (socket says 172.21.15.32:35320) 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.639+0000 7f0632ffd640 1 -- 172.21.15.32:0/1399666295 learned_addr learned my addr 172.21.15.32:0/1399666295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0633fff640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103bc0 0x7f063419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f06337fe640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 0x7f06341981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 -- 172.21.15.32:0/1399666295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103bc0 msgr2=0x7f063419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103bc0 0x7f063419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 -- 172.21.15.32:0/1399666295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 msgr2=0x7f06341981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 0x7f06341981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 -- 172.21.15.32:0/1399666295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063419fde0 con 0x7f0634103200 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f06337fe640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 0x7f06341981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0632ffd640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f062400c970 tx=0x7f062400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624007bb0 con 0x7f0634103200 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0624007d50 con 0x7f0634103200 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06341a00d0 con 0x7f0634103200 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062400f450 con 0x7f0634103200 2024-04-23T18:14:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.640+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341a05d0 con 0x7f0634103200 2024-04-23T18:14:29.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.641+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06341093b0 con 0x7f0634103200 2024-04-23T18:14:29.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.642+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0624004f20 con 0x7f0634103200 2024-04-23T18:14:29.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.643+0000 7f0630ff9640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 0x7f061c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.643+0000 7f06337fe640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 0x7f061c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.643+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f06240d01a0 con 0x7f0634103200 2024-04-23T18:14:29.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.644+0000 7f06337fe640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 0x7f061c0a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0628000c00 tx=0x7f06280314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:29.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.648+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0624014030 con 0x7f0634103200 2024-04-23T18:14:29.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.836+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0634062b50 con 0x7f0634103200 2024-04-23T18:14:29.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.838+0000 7f0630ff9640 1 -- 172.21.15.32:0/1399666295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7f062409cb50 con 0x7f0634103200 2024-04-23T18:14:29.832 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:29.832 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.839+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 msgr2=0x7f061c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.839+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 0x7f061c0a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0628000c00 tx=0x7f06280314a0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.839+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 msgr2=0x7f0634198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f062400c970 tx=0x7f062400ce30 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 shutdown_connections 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0634103bc0 0x7f063419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0634103200 0x7f0634198700 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f061c0a3bb0 0x7f061c0a6070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 --2- 172.21.15.32:0/1399666295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0634102010 0x7f06341981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 >> 172.21.15.32:0/1399666295 conn(0x7f06340fd580 msgr2=0x7f0634106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 shutdown_connections 2024-04-23T18:14:29.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.840+0000 7f063a059640 1 -- 172.21.15.32:0/1399666295 wait complete. 2024-04-23T18:14:29.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:29.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:14:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 -- 172.21.15.32:0/2874871198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 msgr2=0x7f453c107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 --2- 172.21.15.32:0/2874871198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c107740 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4530009900 tx=0x7f453002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 -- 172.21.15.32:0/2874871198 shutdown_connections 2024-04-23T18:14:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 --2- 172.21.15.32:0/2874871198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f453c107c80 0x7f453c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 --2- 172.21.15.32:0/2874871198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c107740 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 --2- 172.21.15.32:0/2874871198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 0x7f453c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 -- 172.21.15.32:0/2874871198 >> 172.21.15.32:0/2874871198 conn(0x7f453c0fc430 msgr2=0x7f453c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 -- 172.21.15.32:0/2874871198 shutdown_connections 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.957+0000 7f4542fcd640 1 -- 172.21.15.32:0/2874871198 wait complete. 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 Processor -- start 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 -- start start 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 0x7f453c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f453c107c80 0x7f453c19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f453c10dbf0 con 0x7f453c105350 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f453c10da70 con 0x7f453c102a20 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f4542fcd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f453c10dd70 con 0x7f453c107c80 2024-04-23T18:14:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f453bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f453bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35330/0 (socket says 172.21.15.32:35330) 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.958+0000 7f453bfff640 1 -- 172.21.15.32:0/3169624963 learned_addr learned my addr 172.21.15.32:0/3169624963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 -- 172.21.15.32:0/3169624963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f453c107c80 msgr2=0x7f453c19f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f453c107c80 0x7f453c19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 -- 172.21.15.32:0/3169624963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 msgr2=0x7f453c198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4540d69640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 0x7f453c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 0x7f453c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 -- 172.21.15.32:0/3169624963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c19ff70 con 0x7f453c105350 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f453bfff640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4530031500 tx=0x7f4530003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453003f070 con 0x7f453c105350 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f453003b460 con 0x7f453c105350 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453c1a0200 con 0x7f453c105350 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4530035030 con 0x7f453c105350 2024-04-23T18:14:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.959+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453c1a07c0 con 0x7f453c105350 2024-04-23T18:14:29.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.960+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c0fdf20 con 0x7f453c105350 2024-04-23T18:14:29.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.961+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4530042070 con 0x7f453c105350 2024-04-23T18:14:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.962+0000 7f4539ffb640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 0x7f45240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.962+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f45300f0e00 con 0x7f453c105350 2024-04-23T18:14:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.962+0000 7f4540d69640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 0x7f45240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.962+0000 7f4540d69640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 0x7f45240a60f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4534007dc0 tx=0x7f45340072f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:29.966+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45300bd7b0 con 0x7f453c105350 2024-04-23T18:14:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.152+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f453c062b50 con 0x7f453c105350 2024-04-23T18:14:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.152+0000 7f4539ffb640 1 -- 172.21.15.32:0/3169624963 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f45300bd0b0 con 0x7f453c105350 2024-04-23T18:14:30.147 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:14:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 msgr2=0x7f45240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 0x7f45240a60f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4534007dc0 tx=0x7f45340072f0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 msgr2=0x7f453c198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4530031500 tx=0x7f4530003d50 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 shutdown_connections 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f453c107c80 0x7f453c19f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f453c105350 0x7f453c198890 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f45240a3c30 0x7f45240a60f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 --2- 172.21.15.32:0/3169624963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f453c102a20 0x7f453c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.154+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 >> 172.21.15.32:0/3169624963 conn(0x7f453c0fc430 msgr2=0x7f453c103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.155+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 shutdown_connections 2024-04-23T18:14:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.155+0000 7f4542fcd640 1 -- 172.21.15.32:0/3169624963 wait complete. 2024-04-23T18:14:30.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:29 smithi032 ceph-mon[29221]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:14:30.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:29 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1399666295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:30.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:14:30.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:29 smithi064 ceph-mon[33062]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:14:30.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:29 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1399666295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- 172.21.15.32:0/4043369389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448101fc0 msgr2=0x7fd448102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 --2- 172.21.15.32:0/4043369389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448101fc0 0x7fd448102440 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd438009900 tx=0x7fd43802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- 172.21.15.32:0/4043369389 shutdown_connections 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 --2- 172.21.15.32:0/4043369389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448102980 0x7fd4481092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 --2- 172.21.15.32:0/4043369389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448101fc0 0x7fd448102440 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 --2- 172.21.15.32:0/4043369389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448100e70 0x7fd448101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- 172.21.15.32:0/4043369389 >> 172.21.15.32:0/4043369389 conn(0x7fd4480fc3c0 msgr2=0x7fd4480fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- 172.21.15.32:0/4043369389 shutdown_connections 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- 172.21.15.32:0/4043369389 wait complete. 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 Processor -- start 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.271+0000 7fd44fce1640 1 -- start start 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448101fc0 0x7fd4481975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 0x7fd44819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd44810c9a0 con 0x7fd448100e70 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd44810c820 con 0x7fd448101fc0 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd44810cb20 con 0x7fd448102980 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35356/0 (socket says 172.21.15.32:35356) 2024-04-23T18:14:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 -- 172.21.15.32:0/3521105215 learned_addr learned my addr 172.21.15.32:0/3521105215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44e27e640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 0x7fd44819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 -- 172.21.15.32:0/3521105215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 msgr2=0x7fd44819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44d27c640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448101fc0 0x7fd4481975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 0x7fd44819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 -- 172.21.15.32:0/3521105215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448101fc0 msgr2=0x7fd4481975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448101fc0 0x7fd4481975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 -- 172.21.15.32:0/3521105215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44819ec90 con 0x7fd448100e70 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44e27e640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 0x7fd44819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44da7d640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d950 tx=0x7fd44400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444017070 con 0x7fd448100e70 2024-04-23T18:14:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd444009c30 con 0x7fd448100e70 2024-04-23T18:14:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44819ef80 con 0x7fd448100e70 2024-04-23T18:14:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44400f620 con 0x7fd448100e70 2024-04-23T18:14:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.272+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44819f3e0 con 0x7fd448100e70 2024-04-23T18:14:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.273+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd448108220 con 0x7fd448100e70 2024-04-23T18:14:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.275+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd444010460 con 0x7fd448100e70 2024-04-23T18:14:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.275+0000 7fd43effd640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.275+0000 7fd44d27c640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.275+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd4440d0090 con 0x7fd448100e70 2024-04-23T18:14:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.276+0000 7fd44d27c640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 0x7fd4340a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd438031500 tx=0x7fd438000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.279+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd44409caa0 con 0x7fd448100e70 2024-04-23T18:14:30.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:29 smithi040 ceph-mon[32626]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:14:30.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:29 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1399666295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.466+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd44802d830 con 0x7fd448100e70 2024-04-23T18:14:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.466+0000 7fd43effd640 1 -- 172.21.15.32:0/3521105215 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fd444009de0 con 0x7fd448100e70 2024-04-23T18:14:30.460 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 msgr2=0x7fd4340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 0x7fd4340a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd438031500 tx=0x7fd438000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 msgr2=0x7fd448197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d950 tx=0x7fd44400de10 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 shutdown_connections 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd448102980 0x7fd44819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.468+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd448101fc0 0x7fd4481975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.469+0000 7fd44fce1640 1 --2- 172.21.15.32:0/3521105215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd448100e70 0x7fd448197070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.469+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 >> 172.21.15.32:0/3521105215 conn(0x7fd4480fc3c0 msgr2=0x7fd4481053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.469+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 shutdown_connections 2024-04-23T18:14:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.469+0000 7fd44fce1640 1 -- 172.21.15.32:0/3521105215 wait complete. 2024-04-23T18:14:30.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:14:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- 172.21.15.32:0/2712954159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798100e90 msgr2=0x7f27981012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 --2- 172.21.15.32:0/2712954159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798100e90 0x7f27981012b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2788009ca0 tx=0x7f278802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- 172.21.15.32:0/2712954159 shutdown_connections 2024-04-23T18:14:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 --2- 172.21.15.32:0/2712954159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27981029a0 0x7f27981092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 --2- 172.21.15.32:0/2712954159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2798101fe0 0x7f2798102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 --2- 172.21.15.32:0/2712954159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798100e90 0x7f27981012b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- 172.21.15.32:0/2712954159 >> 172.21.15.32:0/2712954159 conn(0x7f27980fc420 msgr2=0x7f27980fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- 172.21.15.32:0/2712954159 shutdown_connections 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- 172.21.15.32:0/2712954159 wait complete. 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 Processor -- start 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.585+0000 7f279f449640 1 -- start start 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 0x7f2798069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27981029a0 0x7f279819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f279810c9e0 con 0x7f2798101fe0 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f279810c860 con 0x7f27981029a0 2024-04-23T18:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f279810cb60 con 0x7f2798100e90 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35374/0 (socket says 172.21.15.32:35374) 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 -- 172.21.15.32:0/3837369087 learned_addr learned my addr 172.21.15.32:0/3837369087 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279d1e5640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 0x7f2798069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279d9e6640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27981029a0 0x7f279819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 -- 172.21.15.32:0/3837369087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 msgr2=0x7f2798069d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 0x7f2798069d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 -- 172.21.15.32:0/3837369087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27981029a0 msgr2=0x7f279819f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27981029a0 0x7f279819f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 -- 172.21.15.32:0/3837369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279819fe20 con 0x7f2798101fe0 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279d1e5640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 0x7f2798069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279c9e4640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f278c00d950 tx=0x7f278c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c017020 con 0x7f2798101fe0 2024-04-23T18:14:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27981a0110 con 0x7f2798101fe0 2024-04-23T18:14:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f278c009c30 con 0x7f2798101fe0 2024-04-23T18:14:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c00f6b0 con 0x7f2798101fe0 2024-04-23T18:14:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.586+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981a06b0 con 0x7f2798101fe0 2024-04-23T18:14:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.587+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2798108310 con 0x7f2798101fe0 2024-04-23T18:14:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.588+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f278c01a020 con 0x7f2798101fe0 2024-04-23T18:14:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.589+0000 7f27967fc640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 0x7f27800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.589+0000 7f279d1e5640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 0x7f27800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.589+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f278c0d04a0 con 0x7f2798101fe0 2024-04-23T18:14:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.590+0000 7f279d1e5640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 0x7f27800a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2788031b50 tx=0x7f2788005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.593+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f278c009de0 con 0x7f2798101fe0 2024-04-23T18:14:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.778+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2798062b50 con 0x7f2798101fe0 2024-04-23T18:14:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.779+0000 7f27967fc640 1 -- 172.21.15.32:0/3837369087 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f278c09ceb0 con 0x7f2798101fe0 2024-04-23T18:14:30.773 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:14:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 msgr2=0x7f27800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 0x7f27800a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2788031b50 tx=0x7f2788005ef0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 msgr2=0x7f279819cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f278c00d950 tx=0x7f278c00de10 comp rx=0 tx=0).stop 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 shutdown_connections 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f27800a3bb0 0x7f27800a6070 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27981029a0 0x7f279819f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2798101fe0 0x7f279819cf90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 --2- 172.21.15.32:0/3837369087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2798100e90 0x7f2798069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 >> 172.21.15.32:0/3837369087 conn(0x7f27980fc420 msgr2=0x7f27981054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 shutdown_connections 2024-04-23T18:14:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.781+0000 7f279f449640 1 -- 172.21.15.32:0/3837369087 wait complete. 2024-04-23T18:14:30.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 -- 172.21.15.32:0/3831879533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 msgr2=0x7faea41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 --2- 172.21.15.32:0/3831879533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea41023d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009900 tx=0x7fae8c02a410 comp rx=0 tx=0).stop 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 -- 172.21.15.32:0/3831879533 shutdown_connections 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 --2- 172.21.15.32:0/3831879533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 0x7faea410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 --2- 172.21.15.32:0/3831879533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 0x7faea4103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 --2- 172.21.15.32:0/3831879533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea41023d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 -- 172.21.15.32:0/3831879533 >> 172.21.15.32:0/3831879533 conn(0x7faea40fd540 msgr2=0x7faea40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 -- 172.21.15.32:0/3831879533 shutdown_connections 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.895+0000 7faeaa805640 1 -- 172.21.15.32:0/3831879533 wait complete. 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 Processor -- start 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 -- start start 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 0x7faea4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 0x7faea4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faea410daf0 con 0x7faea4101fb0 2024-04-23T18:14:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d970 con 0x7faea41031a0 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faeaa805640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faea410dc70 con 0x7faea4103b60 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35396/0 (socket says 172.21.15.32:35396) 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 -- 172.21.15.32:0/2266311019 learned_addr learned my addr 172.21.15.32:0/2266311019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea8da2640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 0x7faea4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea37fe640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 0x7faea4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 -- 172.21.15.32:0/2266311019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 msgr2=0x7faea4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 0x7faea4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 -- 172.21.15.32:0/2266311019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 msgr2=0x7faea4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 0x7faea4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.896+0000 7faea3fff640 1 -- 172.21.15.32:0/2266311019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea4075110 con 0x7faea4101fb0 2024-04-23T18:14:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.897+0000 7faea3fff640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006010 tx=0x7fae8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.898+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c03f030 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.898+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae8c003cb0 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.898+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea41a2210 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.898+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c035070 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.898+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea41a2760 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.899+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea4109350 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.900+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fae8c042030 con 0x7faea4101fb0 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.901+0000 7faea17fa640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.901+0000 7faea37fe640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.901+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fae8c0f09c0 con 0x7faea4101fb0 2024-04-23T18:14:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.901+0000 7faea37fe640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 0x7fae900a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faea4075a70 tx=0x7fae9400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:30.905+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae8c0bcd20 con 0x7faea4101fb0 2024-04-23T18:14:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.132+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faea402d5c0 con 0x7faea4101fb0 2024-04-23T18:14:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.133+0000 7faea17fa640 1 -- 172.21.15.32:0/2266311019 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fae8c0c1020 con 0x7faea4101fb0 2024-04-23T18:14:31.127 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:31.127 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:14:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 msgr2=0x7fae900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 0x7fae900a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faea4075a70 tx=0x7fae9400b410 comp rx=0 tx=0).stop 2024-04-23T18:14:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 msgr2=0x7faea4075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006010 tx=0x7fae8c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 shutdown_connections 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faea4103b60 0x7faea4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faea41031a0 0x7faea4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 --2- 172.21.15.32:0/2266311019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faea4101fb0 0x7faea4075d00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 >> 172.21.15.32:0/2266311019 conn(0x7faea40fd540 msgr2=0x7faea4106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 shutdown_connections 2024-04-23T18:14:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.135+0000 7faeaa805640 1 -- 172.21.15.32:0/2266311019 wait complete. 2024-04-23T18:14:31.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:30 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3169624963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:30 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3521105215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:30 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3837369087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:30 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3169624963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:30 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3521105215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:30 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3837369087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 -- 172.21.15.32:0/3168783936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105dc0 msgr2=0x7f3f3010c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 --2- 172.21.15.32:0/3168783936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105dc0 0x7f3f3010c650 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009ca0 tx=0x7f3f2802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 -- 172.21.15.32:0/3168783936 shutdown_connections 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 --2- 172.21.15.32:0/3168783936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105dc0 0x7f3f3010c650 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 --2- 172.21.15.32:0/3168783936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105400 0x7f3f30105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 --2- 172.21.15.32:0/3168783936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 0x7f3f30104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 -- 172.21.15.32:0/3168783936 >> 172.21.15.32:0/3168783936 conn(0x7f3f300ff7a0 msgr2=0x7f3f30101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 -- 172.21.15.32:0/3168783936 shutdown_connections 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.250+0000 7f3f34891640 1 -- 172.21.15.32:0/3168783936 wait complete. 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 Processor -- start 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 -- start start 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 0x7f3f3019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105dc0 0x7f3f301a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3010fd20 con 0x7f3f30105400 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3010fba0 con 0x7f3f30105dc0 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f34891640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3010fea0 con 0x7f3f30104210 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35402/0 (socket says 172.21.15.32:35402) 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 -- 172.21.15.32:0/1274844482 learned_addr learned my addr 172.21.15.32:0/1274844482 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f2e59c640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 0x7f3f3019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f2ed9d640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105dc0 0x7f3f301a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 -- 172.21.15.32:0/1274844482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 msgr2=0x7f3f3019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 0x7f3f3019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 -- 172.21.15.32:0/1274844482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105dc0 msgr2=0x7f3f301a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105dc0 0x7f3f301a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 -- 172.21.15.32:0/1274844482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f301a1f50 con 0x7f3f30105400 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.251+0000 7f3f27fff640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800b4a0 tx=0x7f3f1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.252+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18014070 con 0x7f3f30105400 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.252+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f18010ac0 con 0x7f3f30105400 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.252+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f301a2240 con 0x7f3f30105400 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.252+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18017c30 con 0x7f3f30105400 2024-04-23T18:14:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.252+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f301a2810 con 0x7f3f30105400 2024-04-23T18:14:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.253+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3010b4d0 con 0x7f3f30105400 2024-04-23T18:14:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.254+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3f18017070 con 0x7f3f30105400 2024-04-23T18:14:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.254+0000 7f3f277fe640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 0x7f3f1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.255+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3f180cfa50 con 0x7f3f30105400 2024-04-23T18:14:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.255+0000 7f3f2e59c640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 0x7f3f1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.255+0000 7f3f2e59c640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 0x7f3f1c0a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f30105260 tx=0x7f3f20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.258+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f1809be10 con 0x7f3f30105400 2024-04-23T18:14:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:30 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3169624963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:30 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3521105215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:30 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3837369087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.448+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f3002d830 con 0x7f3f30105400 2024-04-23T18:14:31.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.449+0000 7f3f277fe640 1 -- 172.21.15.32:0/1274844482 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7f3f180a0110 con 0x7f3f30105400 2024-04-23T18:14:31.444 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:31.444 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 msgr2=0x7f3f1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 0x7f3f1c0a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f30105260 tx=0x7f3f20005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 msgr2=0x7f3f3019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800b4a0 tx=0x7f3f1800b960 comp rx=0 tx=0).stop 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 shutdown_connections 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f1c0a3bb0 0x7f3f1c0a6070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f30105dc0 0x7f3f301a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f30105400 0x7f3f3019a870 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 --2- 172.21.15.32:0/1274844482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f30104210 0x7f3f3019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.451+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 >> 172.21.15.32:0/1274844482 conn(0x7f3f300ff7a0 msgr2=0x7f3f301086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.452+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 shutdown_connections 2024-04-23T18:14:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.452+0000 7f3f34891640 1 -- 172.21.15.32:0/1274844482 wait complete. 2024-04-23T18:14:31.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:31.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:14:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 -- 172.21.15.32:0/72987900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198102a20 msgr2=0x7f1198104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 --2- 172.21.15.32:0/72987900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198102a20 0x7f1198104e10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1194009900 tx=0x7f119402a410 comp rx=0 tx=0).stop 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 -- 172.21.15.32:0/72987900 shutdown_connections 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 --2- 172.21.15.32:0/72987900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1198107c80 0x7f119810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 --2- 172.21.15.32:0/72987900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198105350 0x7f1198107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 --2- 172.21.15.32:0/72987900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198102a20 0x7f1198104e10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 -- 172.21.15.32:0/72987900 >> 172.21.15.32:0/72987900 conn(0x7f11980fc430 msgr2=0x7f11980fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 -- 172.21.15.32:0/72987900 shutdown_connections 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.569+0000 7f119ffbb640 1 -- 172.21.15.32:0/72987900 wait complete. 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 Processor -- start 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 -- start start 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1198102a20 0x7f1198198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198107c80 0x7f119819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f119810dae0 con 0x7f1198105350 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f119810d960 con 0x7f1198107c80 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119ffbb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f119810dc60 con 0x7f1198102a20 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35414/0 (socket says 172.21.15.32:35414) 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 -- 172.21.15.32:0/3263838631 learned_addr learned my addr 172.21.15.32:0/3263838631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119e558640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198107c80 0x7f119819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 -- 172.21.15.32:0/3263838631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1198102a20 msgr2=0x7f1198198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1198102a20 0x7f1198198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 -- 172.21.15.32:0/3263838631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198107c80 msgr2=0x7f119819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198107c80 0x7f119819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 -- 172.21.15.32:0/3263838631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119819fe50 con 0x7f1198105350 2024-04-23T18:14:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.570+0000 7f119d556640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f118800d770 tx=0x7f118800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.571+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188017070 con 0x7f1198105350 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.571+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1188009c30 con 0x7f1198105350 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.571+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11981a0140 con 0x7f1198105350 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.571+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118800f620 con 0x7f1198105350 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.571+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11981a0690 con 0x7f1198105350 2024-04-23T18:14:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.572+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11980fdf20 con 0x7f1198105350 2024-04-23T18:14:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.573+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f118801a070 con 0x7f1198105350 2024-04-23T18:14:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.573+0000 7f118effd640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 0x7f11840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.573+0000 7f119dd57640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 0x7f11840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.573+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f11880cffe0 con 0x7f1198105350 2024-04-23T18:14:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.574+0000 7f119dd57640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 0x7f11840a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11940096c0 tx=0x7f1194000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.578+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1188009de0 con 0x7f1198105350 2024-04-23T18:14:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.763+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1198062b80 con 0x7f1198105350 2024-04-23T18:14:31.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.764+0000 7f118effd640 1 -- 172.21.15.32:0/3263838631 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f118809c9f0 con 0x7f1198105350 2024-04-23T18:14:31.758 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 msgr2=0x7f11840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 0x7f11840a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11940096c0 tx=0x7f1194000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 msgr2=0x7f1198198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f118800d770 tx=0x7f118800dc30 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 shutdown_connections 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11840a3bb0 0x7f11840a6070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1198107c80 0x7f119819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1198105350 0x7f1198198770 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 --2- 172.21.15.32:0/3263838631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1198102a20 0x7f1198198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 >> 172.21.15.32:0/3263838631 conn(0x7f11980fc430 msgr2=0x7f11981032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 shutdown_connections 2024-04-23T18:14:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.766+0000 7f119ffbb640 1 -- 172.21.15.32:0/3263838631 wait complete. 2024-04-23T18:14:31.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- 172.21.15.32:0/2291986684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102a00 msgr2=0x7f9548109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 --2- 172.21.15.32:0/2291986684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102a00 0x7f9548109320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9540009ca0 tx=0x7f954002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- 172.21.15.32:0/2291986684 shutdown_connections 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 --2- 172.21.15.32:0/2291986684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102a00 0x7f9548109320 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 --2- 172.21.15.32:0/2291986684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102040 0x7f95481024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 --2- 172.21.15.32:0/2291986684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- 172.21.15.32:0/2291986684 >> 172.21.15.32:0/2291986684 conn(0x7f95480fc400 msgr2=0x7f95480fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- 172.21.15.32:0/2291986684 shutdown_connections 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- 172.21.15.32:0/2291986684 wait complete. 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 Processor -- start 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.882+0000 7f954fd19640 1 -- start start 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102a00 0x7f954819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f954810daf0 con 0x7f9548102040 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f954810d970 con 0x7f9548100e60 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f954810dc70 con 0x7f9548102a00 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35428/0 (socket says 172.21.15.32:35428) 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 -- 172.21.15.32:0/2833250699 learned_addr learned my addr 172.21.15.32:0/2833250699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954dab5640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954e2b6640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102a00 0x7f954819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 -- 172.21.15.32:0/2833250699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102a00 msgr2=0x7f954819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102a00 0x7f954819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 -- 172.21.15.32:0/2833250699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 msgr2=0x7f9548198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 -- 172.21.15.32:0/2833250699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954819fd60 con 0x7f9548102040 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954dab5640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954d2b4640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f953800c890 tx=0x7f953800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538004220 con 0x7f9548102040 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95380043c0 con 0x7f9548102040 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f954819fff0 con 0x7f9548102040 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538004a00 con 0x7f9548102040 2024-04-23T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.883+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95481a0510 con 0x7f9548102040 2024-04-23T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.884+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9548108200 con 0x7f9548102040 2024-04-23T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.885+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f95380079b0 con 0x7f9548102040 2024-04-23T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.886+0000 7f953effd640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 0x7f95340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.886+0000 7f954dab5640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 0x7f95340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.886+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f95380d01c0 con 0x7f9548102040 2024-04-23T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.887+0000 7f954dab5640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 0x7f95340a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9548101bb0 tx=0x7f954400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:31.890+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9538014030 con 0x7f9548102040 2024-04-23T18:14:32.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.078+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f954802d5c0 con 0x7f9548102040 2024-04-23T18:14:32.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.079+0000 7f953effd640 1 -- 172.21.15.32:0/2833250699 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f953809cbd0 con 0x7f9548102040 2024-04-23T18:14:32.073 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 msgr2=0x7f95340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 0x7f95340a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9548101bb0 tx=0x7f954400a560 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 msgr2=0x7f9548198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f953800c890 tx=0x7f953800cd50 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 shutdown_connections 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9548102a00 0x7f954819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9548102040 0x7f9548198680 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f95340a3bb0 0x7f95340a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 --2- 172.21.15.32:0/2833250699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9548100e60 0x7f9548198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 >> 172.21.15.32:0/2833250699 conn(0x7f95480fc400 msgr2=0x7f95481053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 shutdown_connections 2024-04-23T18:14:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.081+0000 7f954fd19640 1 -- 172.21.15.32:0/2833250699 wait complete. 2024-04-23T18:14:32.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:31 smithi032 ceph-mon[29221]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:32.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:31 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2266311019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:31 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1274844482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:31 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3263838631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:32.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:14:32.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:31 smithi064 ceph-mon[33062]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:32.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:31 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2266311019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:31 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1274844482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:31 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3263838631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.196+0000 7fa745b6f640 1 -- 172.21.15.32:0/2392976367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740101fe0 msgr2=0x7fa740102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.196+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2392976367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740101fe0 0x7fa740102460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa734009900 tx=0x7fa73402a410 comp rx=0 tx=0).stop 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 -- 172.21.15.32:0/2392976367 shutdown_connections 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2392976367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa7401092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2392976367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740101fe0 0x7fa740102460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2392976367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740100e90 0x7fa7401012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 -- 172.21.15.32:0/2392976367 >> 172.21.15.32:0/2392976367 conn(0x7fa7400fc420 msgr2=0x7fa7400fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 -- 172.21.15.32:0/2392976367 shutdown_connections 2024-04-23T18:14:32.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 -- 172.21.15.32:0/2392976367 wait complete. 2024-04-23T18:14:32.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 Processor -- start 2024-04-23T18:14:32.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 -- start start 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.197+0000 7fa745b6f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 0x7fa7401988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa74019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010dbd0 con 0x7fa740100e90 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010da50 con 0x7fa740101fe0 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010dd50 con 0x7fa7401029a0 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35448/0 (socket says 172.21.15.32:35448) 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 -- 172.21.15.32:0/2606316236 learned_addr learned my addr 172.21.15.32:0/2606316236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73effd640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 0x7fa7401988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73ffff640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa74019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 -- 172.21.15.32:0/2606316236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 msgr2=0x7fa74019f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa74019f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 -- 172.21.15.32:0/2606316236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 msgr2=0x7fa7401988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 0x7fa7401988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 -- 172.21.15.32:0/2606316236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74019ffc0 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73effd640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 0x7fa7401988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73ffff640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa74019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73f7fe640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa73000b4a0 tx=0x7fa73000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730014070 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa730010ac0 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7401a02b0 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730017c30 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.198+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7401a0880 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.199+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa740108310 con 0x7fa740100e90 2024-04-23T18:14:32.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.200+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa730017070 con 0x7fa740100e90 2024-04-23T18:14:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.201+0000 7fa73cff9640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 0x7fa7280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.201+0000 7fa73effd640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 0x7fa7280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.201+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa7300cfc10 con 0x7fa740100e90 2024-04-23T18:14:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.202+0000 7fa73effd640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 0x7fa7280a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.205+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa73009c5c0 con 0x7fa740100e90 2024-04-23T18:14:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:31 smithi040 ceph-mon[32626]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:31 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2266311019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:31 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1274844482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:31 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3263838631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.391+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa740062b50 con 0x7fa740100e90 2024-04-23T18:14:32.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.391+0000 7fa73cff9640 1 -- 172.21.15.32:0/2606316236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa73009bec0 con 0x7fa740100e90 2024-04-23T18:14:32.385 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:14:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 msgr2=0x7fa7280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 0x7fa7280a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 msgr2=0x7fa7401983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa73000b4a0 tx=0x7fa73000b960 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 shutdown_connections 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7401029a0 0x7fa74019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa7280a3bb0 0x7fa7280a6070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa740101fe0 0x7fa7401988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 --2- 172.21.15.32:0/2606316236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa740100e90 0x7fa7401983a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 >> 172.21.15.32:0/2606316236 conn(0x7fa7400fc420 msgr2=0x7fa7401054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 shutdown_connections 2024-04-23T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.393+0000 7fa745b6f640 1 -- 172.21.15.32:0/2606316236 wait complete. 2024-04-23T18:14:32.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.506+0000 7f7e15ce1640 1 -- 172.21.15.32:0/3590191623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10104230 msgr2=0x7f7e10104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.506+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/3590191623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10104230 0x7f7e10104650 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009900 tx=0x7f7e0002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.506+0000 7f7e15ce1640 1 -- 172.21.15.32:0/3590191623 shutdown_connections 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/3590191623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10105de0 0x7f7e1010c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/3590191623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105420 0x7f7e101058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/3590191623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10104230 0x7f7e10104650 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 -- 172.21.15.32:0/3590191623 >> 172.21.15.32:0/3590191623 conn(0x7f7e100ff7a0 msgr2=0x7f7e10101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 -- 172.21.15.32:0/3590191623 shutdown_connections 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 -- 172.21.15.32:0/3590191623 wait complete. 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 Processor -- start 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 -- start start 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10104230 0x7f7e1019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.507+0000 7f7e15ce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 0x7f7e101a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e15ce1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1010fdc0 con 0x7f7e10105420 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e15ce1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1010fc40 con 0x7f7e10104230 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e15ce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1010ff40 con 0x7f7e10105de0 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35458/0 (socket says 172.21.15.32:35458) 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 -- 172.21.15.32:0/1082083167 learned_addr learned my addr 172.21.15.32:0/1082083167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0ffff640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 0x7f7e101a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0f7fe640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10104230 0x7f7e1019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 -- 172.21.15.32:0/1082083167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 msgr2=0x7f7e101a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 0x7f7e101a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 -- 172.21.15.32:0/1082083167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10104230 msgr2=0x7f7e1019a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10104230 0x7f7e1019a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 -- 172.21.15.32:0/1082083167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e101a2170 con 0x7f7e10105420 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0ffff640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 0x7f7e101a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0effd640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400b4a0 tx=0x7f7e0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04014070 con 0x7f7e10105420 2024-04-23T18:14:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e04010ac0 con 0x7f7e10105420 2024-04-23T18:14:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e101a2460 con 0x7f7e10105420 2024-04-23T18:14:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04017c30 con 0x7f7e10105420 2024-04-23T18:14:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.508+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e101a29e0 con 0x7f7e10105420 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.509+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1010b6c0 con 0x7f7e10105420 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.510+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7e04017070 con 0x7f7e10105420 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.511+0000 7f7e0cff9640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 0x7f7df80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.511+0000 7f7e0f7fe640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 0x7f7df80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.511+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7e040cf820 con 0x7f7e10105420 2024-04-23T18:14:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.511+0000 7f7e0f7fe640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 0x7f7df80a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e0002a930 tx=0x7f7e00000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.515+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e0409c230 con 0x7f7e10105420 2024-04-23T18:14:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.738+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e1002d5c0 con 0x7f7e10105420 2024-04-23T18:14:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.739+0000 7f7e0cff9640 1 -- 172.21.15.32:0/1082083167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7f7e0409bb30 con 0x7f7e10105420 2024-04-23T18:14:32.733 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:32.733 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:14:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 msgr2=0x7f7df80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 0x7f7df80a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e0002a930 tx=0x7f7e00000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 msgr2=0x7f7e1019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400b4a0 tx=0x7f7e0400b960 comp rx=0 tx=0).stop 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 shutdown_connections 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e10105de0 0x7f7e101a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10105420 0x7f7e1019aa90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7df80a3bb0 0x7f7df80a6070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 --2- 172.21.15.32:0/1082083167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e10104230 0x7f7e1019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 >> 172.21.15.32:0/1082083167 conn(0x7f7e100ff7a0 msgr2=0x7f7e10108890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 shutdown_connections 2024-04-23T18:14:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.741+0000 7f7e15ce1640 1 -- 172.21.15.32:0/1082083167 wait complete. 2024-04-23T18:14:32.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 -- 172.21.15.32:0/719888408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68102a30 msgr2=0x7f3f68104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/719888408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68102a30 0x7f3f68104e20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009900 tx=0x7f3f5002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 -- 172.21.15.32:0/719888408 shutdown_connections 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/719888408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68107c90 0x7f3f6810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/719888408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68105360 0x7f3f68107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/719888408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68102a30 0x7f3f68104e20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 -- 172.21.15.32:0/719888408 >> 172.21.15.32:0/719888408 conn(0x7f3f680fc420 msgr2=0x7f3f680fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 -- 172.21.15.32:0/719888408 shutdown_connections 2024-04-23T18:14:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 -- 172.21.15.32:0/719888408 wait complete. 2024-04-23T18:14:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.854+0000 7f3f6d07c640 1 Processor -- start 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 -- start start 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102a30 0x7f3f681023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68107c90 0x7f3f68100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6810db70 con 0x7f3f68105360 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6810d9f0 con 0x7f3f68107c90 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6d07c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6810dcf0 con 0x7f3f68102a30 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35478/0 (socket says 172.21.15.32:35478) 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 -- 172.21.15.32:0/4161130975 learned_addr learned my addr 172.21.15.32:0/4161130975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:32.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f66d9d640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102a30 0x7f3f681023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6759e640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68107c90 0x7f3f68100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 -- 172.21.15.32:0/4161130975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102a30 msgr2=0x7f3f681023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102a30 0x7f3f681023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 -- 172.21.15.32:0/4161130975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68107c90 msgr2=0x7f3f68100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68107c90 0x7f3f68100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 -- 172.21.15.32:0/4161130975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f681017b0 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.855+0000 7f3f6659c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b450 tx=0x7f3f5800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.856+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58015020 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.856+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f681a2360 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.856+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f58011d40 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.856+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800f340 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.856+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f681a2980 con 0x7f3f68105360 2024-04-23T18:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.857+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f680fdf10 con 0x7f3f68105360 2024-04-23T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.858+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3f58018020 con 0x7f3f68105360 2024-04-23T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.858+0000 7f3f5ffff640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 0x7f3f540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.859+0000 7f3f66d9d640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 0x7f3f540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.859+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3f580d03a0 con 0x7f3f68105360 2024-04-23T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.859+0000 7f3f66d9d640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 0x7f3f540a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f500314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:32.862+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f5801f020 con 0x7f3f68105360 2024-04-23T18:14:33.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2833250699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2606316236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1082083167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.054+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f6802d7c0 con 0x7f3f68105360 2024-04-23T18:14:33.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.055+0000 7f3f5ffff640 1 -- 172.21.15.32:0/4161130975 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7f3f5800d040 con 0x7f3f68105360 2024-04-23T18:14:33.050 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:33.050 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 msgr2=0x7f3f540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 0x7f3f540a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f500314a0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 msgr2=0x7f3f681009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b450 tx=0x7f3f5800b910 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 shutdown_connections 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f3f540a3bb0 0x7f3f540a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68107c90 0x7f3f68100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f68105360 0x7f3f681009e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 --2- 172.21.15.32:0/4161130975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102a30 0x7f3f681023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 >> 172.21.15.32:0/4161130975 conn(0x7f3f680fc420 msgr2=0x7f3f68103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 shutdown_connections 2024-04-23T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.058+0000 7f3f6d07c640 1 -- 172.21.15.32:0/4161130975 wait complete. 2024-04-23T18:14:33.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:33.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 -- 172.21.15.32:0/1618802111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 msgr2=0x7fc2b0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 --2- 172.21.15.32:0/1618802111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0101280 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009900 tx=0x7fc2a402a410 comp rx=0 tx=0).stop 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 -- 172.21.15.32:0/1618802111 shutdown_connections 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 --2- 172.21.15.32:0/1618802111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2b0102a00 0x7fc2b0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 --2- 172.21.15.32:0/1618802111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102040 0x7fc2b01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 --2- 172.21.15.32:0/1618802111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0101280 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 -- 172.21.15.32:0/1618802111 >> 172.21.15.32:0/1618802111 conn(0x7fc2b00fc460 msgr2=0x7fc2b00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 -- 172.21.15.32:0/1618802111 shutdown_connections 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.172+0000 7fc2b71af640 1 -- 172.21.15.32:0/1618802111 wait complete. 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 Processor -- start 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 -- start start 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2b0102040 0x7fc2b0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102a00 0x7fc2b019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b010ca40 con 0x7fc2b0100e60 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b010c8c0 con 0x7fc2b0102a00 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b010cbc0 con 0x7fc2b0102040 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35500/0 (socket says 172.21.15.32:35500) 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 -- 172.21.15.32:0/2450159227 learned_addr learned my addr 172.21.15.32:0/2450159227 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b574c640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102a00 0x7fc2b019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 -- 172.21.15.32:0/2450159227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2b0102040 msgr2=0x7fc2b0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2b0102040 0x7fc2b0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 -- 172.21.15.32:0/2450159227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102a00 msgr2=0x7fc2b019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102a00 0x7fc2b019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 -- 172.21.15.32:0/2450159227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b019fdf0 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b4f4b640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4031500 tx=0x7fc2a4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a403f070 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a4003e70 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b01a0080 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.173+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4035030 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.174+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b01a05c0 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.175+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b01a08e0 con 0x7fc2b0100e60 2024-04-23T18:14:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2833250699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2606316236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1082083167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:33.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.176+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc2a4042070 con 0x7fc2b0100e60 2024-04-23T18:14:33.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.176+0000 7fc2adffb640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 0x7fc2a00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.176+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc2a40f0fa0 con 0x7fc2b0100e60 2024-04-23T18:14:33.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.176+0000 7fc2affff640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 0x7fc2a00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.177+0000 7fc2affff640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 0x7fc2a00a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc2b01996e0 tx=0x7fc29c005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.181+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2a40bda50 con 0x7fc2b0100e60 2024-04-23T18:14:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2833250699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2606316236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1082083167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:33.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.405+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b01a0ab0 con 0x7fc2b0100e60 2024-04-23T18:14:33.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.406+0000 7fc2adffb640 1 -- 172.21.15.32:0/2450159227 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fc2a40bd350 con 0x7fc2b0100e60 2024-04-23T18:14:33.400 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:33.400 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:14:33.400 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 msgr2=0x7fc2a00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 0x7fc2a00a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc2b01996e0 tx=0x7fc29c005bf0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 msgr2=0x7fc2b0198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4031500 tx=0x7fc2a4003c80 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 shutdown_connections 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2a00a3c30 0x7fc2a00a60f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2b0102a00 0x7fc2b019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2b0102040 0x7fc2b0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 --2- 172.21.15.32:0/2450159227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b0100e60 0x7fc2b0198240 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 >> 172.21.15.32:0/2450159227 conn(0x7fc2b00fc460 msgr2=0x7fc2b00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 shutdown_connections 2024-04-23T18:14:33.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.408+0000 7fc2b71af640 1 -- 172.21.15.32:0/2450159227 wait complete. 2024-04-23T18:14:33.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:13:05.375853+0000', 'modified': '2024-04-23T18:13:10.043413+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24551, 'mds_1': 24515, 'mds_2': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/669117639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 669117639}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 669117639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/1068324383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1068324383}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1068324383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6837/1625538187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 1625538187}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 1625538187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.64:6839/1242971440', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 1242971440}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 1242971440}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6837/1154751886', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 1154751886}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 1154751886}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2479383965', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2479383965}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2479383965}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24551, 'qdb_cluster': [24499, 24515, 24551]} 2024-04-23T18:14:33.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:14:33.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 -- 172.21.15.32:0/2336826079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028105550 msgr2=0x7f2028107940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 --2- 172.21.15.32:0/2336826079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028105550 0x7f2028107940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f201c009900 tx=0x7f201c02a410 comp rx=0 tx=0).stop 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 -- 172.21.15.32:0/2336826079 shutdown_connections 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 --2- 172.21.15.32:0/2336826079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028107e80 0x7f202810a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 --2- 172.21.15.32:0/2336826079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028105550 0x7f2028107940 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 --2- 172.21.15.32:0/2336826079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028102c20 0x7f2028105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 -- 172.21.15.32:0/2336826079 >> 172.21.15.32:0/2336826079 conn(0x7f20280fc5f0 msgr2=0x7f20280fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 -- 172.21.15.32:0/2336826079 shutdown_connections 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 -- 172.21.15.32:0/2336826079 wait complete. 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.523+0000 7f202d272640 1 Processor -- start 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 -- start start 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028105550 0x7f2028198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028107e80 0x7f202819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f202810dba0 con 0x7f2028102c20 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f202810da20 con 0x7f2028105550 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f202d272640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f202810dd20 con 0x7f2028107e80 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35512/0 (socket says 172.21.15.32:35512) 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 -- 172.21.15.32:0/2249089515 learned_addr learned my addr 172.21.15.32:0/2249089515 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f20277fe640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028107e80 0x7f202819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f20267fc640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028105550 0x7f2028198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 -- 172.21.15.32:0/2249089515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028107e80 msgr2=0x7f202819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028107e80 0x7f202819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 -- 172.21.15.32:0/2249089515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028105550 msgr2=0x7f2028198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028105550 0x7f2028198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 -- 172.21.15.32:0/2249089515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202819fee0 con 0x7f2028102c20 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.524+0000 7f2026ffd640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f201800d770 tx=0x7f201800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.525+0000 7f200ffff640 1 -- 172.21.15.32:0/2249089515 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018017070 con 0x7f2028102c20 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.525+0000 7f200ffff640 1 -- 172.21.15.32:0/2249089515 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20180054d0 con 0x7f2028102c20 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.525+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20281a01d0 con 0x7f2028102c20 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.525+0000 7f200ffff640 1 -- 172.21.15.32:0/2249089515 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018004c30 con 0x7f2028102c20 2024-04-23T18:14:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.525+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281a06a0 con 0x7f2028102c20 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.527+0000 7f200ffff640 1 -- 172.21.15.32:0/2249089515 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2018004e10 con 0x7f2028102c20 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.527+0000 7f202ff85740 1 Processor -- start 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 -- start start 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42effa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4254c50 con 0x55dbb42f1710 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb41f1830 con 0x55dbb42efb80 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f202ff85740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4250c10 con 0x55dbb42f0d50 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35522/0 (socket says 172.21.15.32:35522) 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 -- 172.21.15.32:0/449880952 learned_addr learned my addr 172.21.15.32:0/449880952 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f2026ffd640 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20277fe640 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 -- 172.21.15.32:0/449880952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 msgr2=0x55dbb42f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 -- 172.21.15.32:0/449880952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 msgr2=0x55dbb42effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42effa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f200ffff640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 0x7f20100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f20267fc640 1 -- 172.21.15.32:0/449880952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbb3f672d0 con 0x55dbb42f1710 2024-04-23T18:14:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.528+0000 7f2026ffd640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 0x7f20100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.529+0000 7f20267fc640 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f202806b160 tx=0x7f201c032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9190f843432c964 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.529+0000 7f200ffff640 1 -- 172.21.15.32:0/2249089515 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f20180cfe90 con 0x7f2028102c20 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.529+0000 7f200d7fa640 1 -- 172.21.15.32:0/449880952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c03f070 con 0x55dbb42f1710 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.529+0000 7f200d7fa640 1 -- 172.21.15.32:0/449880952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c0317a0 con 0x55dbb42f1710 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.529+0000 7f200d7fa640 1 -- 172.21.15.32:0/449880952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c035070 con 0x55dbb42f1710 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- 172.21.15.32:0/449880952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 msgr2=0x55dbb42f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f2026ffd640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 0x7f20100a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2018098b70 tx=0x7f20180d3970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f202806b160 tx=0x7f201c032bd0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- 172.21.15.32:0/449880952 shutdown_connections 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb42f7fa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 --2- 172.21.15.32:0/449880952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- 172.21.15.32:0/449880952 >> 172.21.15.32:0/449880952 conn(0x55dbb42eb070 msgr2=0x55dbb42eb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- 172.21.15.32:0/449880952 shutdown_connections 2024-04-23T18:14:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- 172.21.15.32:0/449880952 wait complete. 2024-04-23T18:14:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 -- start start 2024-04-23T18:14:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.530+0000 7f202ff85740 1 Processor -- start 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42fe360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb43058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbb42fcad0 con 0x55dbb42f0d50 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb41ef470 con 0x55dbb42efb80 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbb42fcc50 con 0x55dbb42f1710 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f202ff85740 10 client.0 _ll_register_callbacks cb 0x55dbb4214f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35534/0 (socket says 172.21.15.32:35534) 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 -- 172.21.15.32:0/1948501402 learned_addr learned my addr 172.21.15.32:0/1948501402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20277fe640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20277fe640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb43058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 -- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 msgr2=0x55dbb43058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb43058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 -- 172.21.15.32:0/1948501402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 msgr2=0x55dbb42fe360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42fe360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 -- 172.21.15.32:0/1948501402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbb40316b0 con 0x55dbb42f0d50 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20277fe640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42fe360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f20267fc640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f201c0045e0 tx=0x7f201c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d123169968b940e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f200aff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.531+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c035070 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f201c0035f0 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f201c0317a0 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f201c03f040 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 10 client.15066 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dbb4308c50 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c0317a0 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f201c059900 con 0x55dbb42f0d50 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 10 client.15066 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 10 client.15066 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 20 client.15066 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb430c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 4 client.15066 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.532+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbb4262670 con 0x55dbb430ae10 2024-04-23T18:14:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.533+0000 7f20277fe640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb430c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.533+0000 7f20277fe640 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb430c280 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3c974f7cf1a216b server_cookie=9b9c78808ce701ce in_seq=0 out_seq=0 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.533+0000 7f200aff5640 10 client.15066 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f200aff5640 1 -- 172.21.15.32:0/1948501402 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbb4262670 con 0x55dbb430ae10 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f200aff5640 10 client.15066 handle_command_reply: tid=0 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 client.15066 shutdown 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 msgr2=0x55dbb42fe8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f201c0045e0 tx=0x7f201c004610 comp rx=0 tx=0).stop 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 shutdown_connections 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb430c280 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb43058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f200aff5640 0 client.15066 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f0d50 0x55dbb42fe8a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 --2- 172.21.15.32:0/1948501402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb42fe360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 >> 172.21.15.32:0/1948501402 conn(0x55dbb42eb070 msgr2=0x55dbb42ea1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 shutdown_connections 2024-04-23T18:14:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 1 -- 172.21.15.32:0/1948501402 wait complete. 2024-04-23T18:14:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.534+0000 7f202ff85740 20 client.15066 trim_cache size 0 max 16384 2024-04-23T18:14:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 Processor -- start 2024-04-23T18:14:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 -- start start 2024-04-23T18:14:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb4306330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42fcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430a5c0 con 0x55dbb42f1710 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430ac90 con 0x55dbb42efb80 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f202ff85740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430b360 con 0x55dbb42f0d50 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f20277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f20277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35540/0 (socket says 172.21.15.32:35540) 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.553+0000 7f20277fe640 1 -- 172.21.15.32:0/213955396 learned_addr learned my addr 172.21.15.32:0/213955396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb4306330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20267fc640 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42fcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 -- 172.21.15.32:0/213955396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 msgr2=0x55dbb42fcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42fcff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 -- 172.21.15.32:0/213955396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 msgr2=0x55dbb4306330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb4306330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 -- 172.21.15.32:0/213955396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbb42ff200 con 0x55dbb42f1710 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f20277fe640 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2020012760 tx=0x7f2020004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90a30151390663f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f200cff9640 1 -- 172.21.15.32:0/213955396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202001b040 con 0x55dbb42f1710 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f200cff9640 1 -- 172.21.15.32:0/213955396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2020004cb0 con 0x55dbb42f1710 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.554+0000 7f200cff9640 1 -- 172.21.15.32:0/213955396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020005280 con 0x55dbb42f1710 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- 172.21.15.32:0/213955396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 msgr2=0x55dbb4309fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2020012760 tx=0x7f2020004a40 comp rx=0 tx=0).stop 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- 172.21.15.32:0/213955396 shutdown_connections 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42f1710 0x55dbb4309fa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f0d50 0x55dbb42fcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- 172.21.15.32:0/213955396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42efb80 0x55dbb4306330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- 172.21.15.32:0/213955396 >> 172.21.15.32:0/213955396 conn(0x55dbb42eb070 msgr2=0x55dbb42eb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- 172.21.15.32:0/213955396 shutdown_connections 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- 172.21.15.32:0/213955396 wait complete. 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- start start 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 Processor -- start 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42f0d50 0x55dbb4301a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb4304a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430c100 con 0x55dbb42efb80 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430ba30 con 0x55dbb42f0d50 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbb430c7d0 con 0x55dbb42f1710 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f202ff85740 10 client.0 _ll_register_callbacks cb 0x7f2020009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f20277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f20277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35544/0 (socket says 172.21.15.32:35544) 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.555+0000 7f20277fe640 1 -- 172.21.15.32:0/754779117 learned_addr learned my addr 172.21.15.32:0/754779117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20267fc640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42f0d50 0x55dbb4301a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb4304a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 -- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 msgr2=0x55dbb4304a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb4304a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 -- 172.21.15.32:0/754779117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42f0d50 msgr2=0x55dbb4301a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42f0d50 0x55dbb4301a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 -- 172.21.15.32:0/754779117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbb4305080 con 0x55dbb42efb80 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f20277fe640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f20200129d0 tx=0x7f2020005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b691db00693d581b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020004ab0 con 0x55dbb42efb80 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f20200107a0 con 0x55dbb42efb80 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2020004cb0 con 0x55dbb42efb80 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f20200109a0 con 0x55dbb42efb80 2024-04-23T18:14:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f202ff85740 10 client.15072 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dbb4371270 con 0x55dbb42efb80 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2020004cb0 con 0x55dbb42efb80 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.556+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f202000f040 con 0x55dbb42efb80 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 10 client.15072 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 10 client.15072 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 20 client.15072 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb4374710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 4 client.15072 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "status"}) -- 0x55dbb425c4e0 con 0x55dbb430ae10 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f20267fc640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb4374710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f200aff5640 10 client.15072 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f20267fc640 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb4374710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37a38ccf46559eda server_cookie=248b7e1cf1dbf42b in_seq=0 out_seq=0 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f200aff5640 1 -- 172.21.15.32:0/754779117 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55dbb425c4e0 con 0x55dbb430ae10 2024-04-23T18:14:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f200aff5640 10 client.15072 handle_command_reply: tid=0 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 1 client.15072 shutdown 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 msgr2=0x55dbb4301530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.557+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f20200129d0 tx=0x7f2020005bd0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 shutdown_connections 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x55dbb430ae10 0x55dbb4374710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbb42f1710 0x55dbb4304a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb42f0d50 0x55dbb4301a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f200aff5640 0 client.15072 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 --2- 172.21.15.32:0/754779117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbb42efb80 0x55dbb4301530 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 >> 172.21.15.32:0/754779117 conn(0x55dbb42eb070 msgr2=0x55dbb42e60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 shutdown_connections 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 1 -- 172.21.15.32:0/754779117 wait complete. 2024-04-23T18:14:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202ff85740 20 client.15072 trim_cache size 0 max 16384 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24499, 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 84.519720942999996, 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 101.81020553 2024-04-23T18:14:33.560 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 msgr2=0x7f20100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 0x7f20100a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2018098b70 tx=0x7f20180d3970 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 msgr2=0x7f20281982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f201800d770 tx=0x7f201800dc30 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 shutdown_connections 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2028107e80 0x7f202819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20100a3bb0 0x7f20100a6070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2028105550 0x7f2028198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 --2- 172.21.15.32:0/2249089515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2028102c20 0x7f20281982c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 >> 172.21.15.32:0/2249089515 conn(0x7f20280fc5f0 msgr2=0x7f2028103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 shutdown_connections 2024-04-23T18:14:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.558+0000 7f202d272640 1 -- 172.21.15.32:0/2249089515 wait complete. 2024-04-23T18:14:33.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:14:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- 172.21.15.32:0/3667365007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 msgr2=0x7f476810a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 --2- 172.21.15.32:0/3667365007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476810a520 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4760009ca0 tx=0x7f476002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- 172.21.15.32:0/3667365007 shutdown_connections 2024-04-23T18:14:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 --2- 172.21.15.32:0/3667365007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476810a520 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 --2- 172.21.15.32:0/3667365007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681032d0 0x7f4768103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 --2- 172.21.15.32:0/3667365007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681020e0 0x7f4768102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- 172.21.15.32:0/3667365007 >> 172.21.15.32:0/3667365007 conn(0x7f47680fd650 msgr2=0x7f47680ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- 172.21.15.32:0/3667365007 shutdown_connections 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- 172.21.15.32:0/3667365007 wait complete. 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 Processor -- start 2024-04-23T18:14:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.673+0000 7f476e508640 1 -- start start 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681020e0 0x7f47681981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681032d0 0x7f4768198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f476810dbb0 con 0x7f4768103c90 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f476810da30 con 0x7f47681020e0 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476e508640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f476810dd30 con 0x7f47681032d0 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35562/0 (socket says 172.21.15.32:35562) 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 -- 172.21.15.32:0/879108394 learned_addr learned my addr 172.21.15.32:0/879108394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f4767fff640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681020e0 0x7f47681981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f47677fe640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681032d0 0x7f4768198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 -- 172.21.15.32:0/879108394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681032d0 msgr2=0x7f4768198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681032d0 0x7f4768198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 -- 172.21.15.32:0/879108394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681020e0 msgr2=0x7f47681981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681020e0 0x7f47681981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 -- 172.21.15.32:0/879108394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476819fdf0 con 0x7f4768103c90 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f476caa5640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f47600097b0 tx=0x7f4760003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.674+0000 7f47657fa640 1 -- 172.21.15.32:0/879108394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760004240 con 0x7f4768103c90 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.675+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47681a0080 con 0x7f4768103c90 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.675+0000 7f47657fa640 1 -- 172.21.15.32:0/879108394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47600317c0 con 0x7f4768103c90 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.675+0000 7f47657fa640 1 -- 172.21.15.32:0/879108394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476002fd80 con 0x7f4768103c90 2024-04-23T18:14:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.675+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47681a0620 con 0x7f4768103c90 2024-04-23T18:14:33.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47657fa640 1 -- 172.21.15.32:0/879108394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f47600043e0 con 0x7f4768103c90 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 Processor -- start 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 -- start start 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc02785fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cc026eac50 con 0x55cc02787710 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cc02687830 con 0x55cc02785b80 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4771226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cc026e6c10 con 0x55cc02786d50 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35572/0 (socket says 172.21.15.32:35572) 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47677fe640 1 -- 172.21.15.32:0/3369718416 learned_addr learned my addr 172.21.15.32:0/3369718416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f4767fff640 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47677fe640 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc02785fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.677+0000 7f47657fa640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 0x7f47540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 -- 172.21.15.32:0/3369718416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 msgr2=0x55cc027871d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027871d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 -- 172.21.15.32:0/3369718416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 msgr2=0x55cc02785fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc02785fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 -- 172.21.15.32:0/3369718416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc023fd2d0 con 0x55cc02787710 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f476caa5640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 0x7f47540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47657fa640 1 -- 172.21.15.32:0/879108394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f47600f1b90 con 0x7f4768103c90 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f47677fe640 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f475800bfd0 tx=0x7f475800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1029c6e00d14b26 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f475e7fc640 1 -- 172.21.15.32:0/3369718416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758012d00 con 0x55cc02787710 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f475e7fc640 1 -- 172.21.15.32:0/3369718416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4758010740 con 0x55cc02787710 2024-04-23T18:14:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.678+0000 7f475e7fc640 1 -- 172.21.15.32:0/3369718416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758003850 con 0x55cc02787710 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- 172.21.15.32:0/3369718416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 msgr2=0x55cc0278dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f476caa5640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 0x7f47540a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f47600c6700 tx=0x7f476008da10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f475800bfd0 tx=0x7f475800b650 comp rx=0 tx=0).stop 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- 172.21.15.32:0/3369718416 shutdown_connections 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc0278dfa0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027871d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 --2- 172.21.15.32:0/3369718416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc02785fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- 172.21.15.32:0/3369718416 >> 172.21.15.32:0/3369718416 conn(0x55cc02781070 msgr2=0x55cc02781470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- 172.21.15.32:0/3369718416 shutdown_connections 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- 172.21.15.32:0/3369718416 wait complete. 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 -- start start 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.679+0000 7f4771226740 1 Processor -- start 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027905a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02787710 0x55cc027975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cc0278e720 con 0x55cc02785b80 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cc02685470 con 0x55cc02787710 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cc0278e8a0 con 0x55cc02786d50 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 10 client.0 _ll_register_callbacks cb 0x55cc026aaf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35586/0 (socket says 172.21.15.32:35586) 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 -- 172.21.15.32:0/250307400 learned_addr learned my addr 172.21.15.32:0/250307400 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4767fff640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027905a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 -- 172.21.15.32:0/250307400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 msgr2=0x55cc027905a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027905a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 -- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02787710 msgr2=0x55cc027975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02787710 0x55cc027975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 -- 172.21.15.32:0/250307400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc024c76b0 con 0x55cc02785b80 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4767fff640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027905a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47677fe640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f475800b810 tx=0x7f475800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bf0f81a42c13a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47477fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758005040 con 0x55cc02785b80 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f47580051e0 con 0x55cc02785b80 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4758010740 con 0x55cc02785b80 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47580053e0 con 0x55cc02785b80 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.680+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cc0279a950 con 0x55cc02785b80 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4758010740 con 0x55cc02785b80 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4758026020 con 0x55cc02785b80 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 10 client.15084 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 20 client.15084 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0279cd60 0x55cc0279e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 4 client.15084 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc026f8670 con 0x55cc0279cd60 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.681+0000 7f4767fff640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0279cd60 0x55cc0279e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.682+0000 7f4767fff640 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0279cd60 0x55cc0279e1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da5f7ba67be3621d server_cookie=c7ef445c3b28342a in_seq=0 out_seq=0 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.682+0000 7f47477fe640 10 client.15084 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.682+0000 7f47477fe640 1 -- 172.21.15.32:0/250307400 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc026f8670 con 0x55cc0279cd60 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f47477fe640 10 client.15084 handle_command_reply: tid=0 2024-04-23T18:14:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 client.15084 shutdown 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 msgr2=0x55cc02790060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f475800b810 tx=0x7f475800b540 comp rx=0 tx=0).stop 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 shutdown_connections 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0279cd60 0x55cc0279e1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02787710 0x55cc027975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02786d50 0x55cc027905a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 --2- 172.21.15.32:0/250307400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02785b80 0x55cc02790060 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 >> 172.21.15.32:0/250307400 conn(0x55cc02781070 msgr2=0x55cc027801d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f47477fe640 0 client.15084 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 shutdown_connections 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 1 -- 172.21.15.32:0/250307400 wait complete. 2024-04-23T18:14:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.683+0000 7f4771226740 20 client.15084 trim_cache size 0 max 16384 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 Processor -- start 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 -- start start 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc0277c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02787710 0x55cc0278fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279c510 con 0x55cc02786d50 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279cbe0 con 0x55cc02785b80 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4771226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279d2b0 con 0x55cc02787710 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35602/0 (socket says 172.21.15.32:35602) 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 -- 172.21.15.32:0/2010663894 learned_addr learned my addr 172.21.15.32:0/2010663894 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 -- 172.21.15.32:0/2010663894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02787710 msgr2=0x55cc0278fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02787710 0x55cc0278fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 -- 172.21.15.32:0/2010663894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 msgr2=0x55cc0277c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc0277c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.701+0000 7f4767fff640 1 -- 172.21.15.32:0/2010663894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0278e7e0 con 0x55cc02786d50 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4767fff640 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55cc02781f40 tx=0x7f4750011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41be3dc15a98e6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f475dffb640 1 -- 172.21.15.32:0/2010663894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475001d5b0 con 0x55cc02786d50 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f475dffb640 1 -- 172.21.15.32:0/2010663894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4750023070 con 0x55cc02786d50 2024-04-23T18:14:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f475dffb640 1 -- 172.21.15.32:0/2010663894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750010430 con 0x55cc02786d50 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 -- 172.21.15.32:0/2010663894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 msgr2=0x55cc0279bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55cc02781f40 tx=0x7f4750011810 comp rx=0 tx=0).stop 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 -- 172.21.15.32:0/2010663894 shutdown_connections 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02787710 0x55cc0278fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02786d50 0x55cc0279bca0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 --2- 172.21.15.32:0/2010663894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02785b80 0x55cc0277c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 -- 172.21.15.32:0/2010663894 >> 172.21.15.32:0/2010663894 conn(0x55cc02781070 msgr2=0x55cc02789d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 -- 172.21.15.32:0/2010663894 shutdown_connections 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.702+0000 7f4771226740 1 -- 172.21.15.32:0/2010663894 wait complete. 2024-04-23T18:14:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 -- start start 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 Processor -- start 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02785b80 0x55cc0278c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02786d50 0x55cc0278cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279e050 con 0x55cc02787710 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279d980 con 0x55cc02786d50 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cc0279e720 con 0x55cc02785b80 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4771226740 10 client.0 _ll_register_callbacks cb 0x7f475800bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f47677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35606/0 (socket says 172.21.15.32:35606) 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f47677fe640 1 -- 172.21.15.32:0/4022914687 learned_addr learned my addr 172.21.15.32:0/4022914687 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f4767fff640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02786d50 0x55cc0278cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.703+0000 7f47677fe640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02785b80 0x55cc0278c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 -- 172.21.15.32:0/4022914687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02785b80 msgr2=0x55cc0278c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02785b80 0x55cc0278c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 -- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02786d50 msgr2=0x55cc0278cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02786d50 0x55cc0278cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 -- 172.21.15.32:0/4022914687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc02788ba0 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47677fe640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f47580057a0 tx=0x7f475800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaa4923a0a80885 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47580107c0 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f475800b410 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4750023070 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4758003ca0 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 10 client.15090 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cc02789ea0 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4750023070 con 0x55cc02787710 2024-04-23T18:14:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4758026070 con 0x55cc02787710 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 10 client.15090 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 10 client.15090 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 20 client.15090 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0280b1a0 0x55cc02807cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 4 client.15090 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.704+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "status"}) -- 0x55cc026f24e0 con 0x55cc0280b1a0 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f4767fff640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0280b1a0 0x55cc02807cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f4767fff640 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0280b1a0 0x55cc02807cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b22f711ec89c53ed server_cookie=142b7b9294ce0630 in_seq=0 out_seq=0 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f47477fe640 10 client.15090 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f47477fe640 1 -- 172.21.15.32:0/4022914687 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cc026f24e0 con 0x55cc0280b1a0 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f47477fe640 10 client.15090 handle_command_reply: tid=0 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f4771226740 1 client.15090 shutdown 2024-04-23T18:14:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 msgr2=0x55cc02788530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.705+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f47580057a0 tx=0x7f475800bda0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 shutdown_connections 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x55cc0280b1a0 0x55cc02807cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cc02787710 0x55cc02788530 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cc02786d50 0x55cc0278cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 --2- 172.21.15.32:0/4022914687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc02785b80 0x55cc0278c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f47477fe640 0 client.15090 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 >> 172.21.15.32:0/4022914687 conn(0x55cc02781070 msgr2=0x55cc0279bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 shutdown_connections 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 1 -- 172.21.15.32:0/4022914687 wait complete. 2024-04-23T18:14:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f4771226740 20 client.15090 trim_cache size 0 max 16384 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24515, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 85.677120947999995, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 104.918191978 2024-04-23T18:14:33.707 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 msgr2=0x7f47540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 0x7f47540a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f47600c6700 tx=0x7f476008da10 comp rx=0 tx=0).stop 2024-04-23T18:14:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 msgr2=0x7f476819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f47600097b0 tx=0x7f4760003c20 comp rx=0 tx=0).stop 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 shutdown_connections 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4768103c90 0x7f476819f780 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47681032d0 0x7f4768198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f47540a3bb0 0x7f47540a6070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 --2- 172.21.15.32:0/879108394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47681020e0 0x7f47681981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 >> 172.21.15.32:0/879108394 conn(0x7f47680fd650 msgr2=0x7f47681065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 shutdown_connections 2024-04-23T18:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.706+0000 7f476e508640 1 -- 172.21.15.32:0/879108394 wait complete. 2024-04-23T18:14:33.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 -- 172.21.15.32:0/1994187727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c1054f0 msgr2=0x7f206c1078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 --2- 172.21.15.32:0/1994187727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c1054f0 0x7f206c1078e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2060009900 tx=0x7f206002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 -- 172.21.15.32:0/1994187727 shutdown_connections 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 --2- 172.21.15.32:0/1994187727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c107e20 0x7f206c10a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 --2- 172.21.15.32:0/1994187727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c1054f0 0x7f206c1078e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 --2- 172.21.15.32:0/1994187727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 0x7f206c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 -- 172.21.15.32:0/1994187727 >> 172.21.15.32:0/1994187727 conn(0x7f206c0fc5f0 msgr2=0x7f206c0fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 -- 172.21.15.32:0/1994187727 shutdown_connections 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.820+0000 7f2070b22640 1 -- 172.21.15.32:0/1994187727 wait complete. 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 Processor -- start 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 -- start start 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 0x7f206c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c1054f0 0x7f206c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f206c10db60 con 0x7f206c107e20 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f206c10d9e0 con 0x7f206c1054f0 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2070b22640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f206c10dce0 con 0x7f206c102bc0 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35630/0 (socket says 172.21.15.32:35630) 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 -- 172.21.15.32:0/952589459 learned_addr learned my addr 172.21.15.32:0/952589459 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206a59c640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 0x7f206c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f2069d9b640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c1054f0 0x7f206c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 -- 172.21.15.32:0/952589459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 msgr2=0x7f206c1982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 0x7f206c1982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 -- 172.21.15.32:0/952589459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c1054f0 msgr2=0x7f206c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c1054f0 0x7f206c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.821+0000 7f206ad9d640 1 -- 172.21.15.32:0/952589459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c19fec0 con 0x7f206c107e20 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f206ad9d640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f206400eaa0 tx=0x7f206400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f205b7fe640 1 -- 172.21.15.32:0/952589459 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206400cdb0 con 0x7f206c107e20 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c1a01b0 con 0x7f206c107e20 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f205b7fe640 1 -- 172.21.15.32:0/952589459 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2064005020 con 0x7f206c107e20 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f205b7fe640 1 -- 172.21.15.32:0/952589459 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20640116d0 con 0x7f206c107e20 2024-04-23T18:14:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.822+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c1a0680 con 0x7f206c107e20 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f205b7fe640 1 -- 172.21.15.32:0/952589459 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f20640051d0 con 0x7f206c107e20 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 Processor -- start 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 -- start start 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b048413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b048493b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b047a1c50 con 0x558b04842160 2024-04-23T18:14:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b0473e830 con 0x558b04840f90 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2073831740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558b0479dc10 con 0x558b04842b20 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35634/0 (socket says 172.21.15.32:35634) 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f206a59c640 1 -- 172.21.15.32:0/3584080197 learned_addr learned my addr 172.21.15.32:0/3584080197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f205b7fe640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 0x7f20540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.825+0000 7f2069d9b640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b048413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f2069d9b640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b048493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206ad9d640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 0x7f20540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 -- 172.21.15.32:0/3584080197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 msgr2=0x558b048493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b048493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 -- 172.21.15.32:0/3584080197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 msgr2=0x558b048413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b048413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 -- 172.21.15.32:0/3584080197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b044b42d0 con 0x558b04842160 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f2069d9b640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b048413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f205b7fe640 1 -- 172.21.15.32:0/952589459 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f20640d0520 con 0x7f206c107e20 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206a59c640 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f205c00bd10 tx=0x7f205c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ccb56d2e03e294 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f2058ff9640 1 -- 172.21.15.32:0/3584080197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c004030 con 0x558b04842160 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f2058ff9640 1 -- 172.21.15.32:0/3584080197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c0035f0 con 0x558b04842160 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f206ad9d640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 0x7f20540a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f20640ce3c0 tx=0x7f2064066fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.826+0000 7f2058ff9640 1 -- 172.21.15.32:0/3584080197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c0117b0 con 0x558b04842160 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- 172.21.15.32:0/3584080197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 msgr2=0x558b048425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f205c00bd10 tx=0x7f205c0107a0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- 172.21.15.32:0/3584080197 shutdown_connections 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b048493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b048425e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- 172.21.15.32:0/3584080197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b048413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- 172.21.15.32:0/3584080197 >> 172.21.15.32:0/3584080197 conn(0x558b04838070 msgr2=0x558b04838470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- 172.21.15.32:0/3584080197 shutdown_connections 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- 172.21.15.32:0/3584080197 wait complete. 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- start start 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 Processor -- start 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04840f90 0x558b0484b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04842b20 0x558b04852990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b04849b30 con 0x558b04842160 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b0473c470 con 0x558b04842b20 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.827+0000 7f2073831740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558b04849cb0 con 0x558b04840f90 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2073831740 10 client.0 _ll_register_callbacks cb 0x558b04761f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35644/0 (socket says 172.21.15.32:35644) 2024-04-23T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 -- 172.21.15.32:0/1536233826 learned_addr learned my addr 172.21.15.32:0/1536233826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2069d9b640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04840f90 0x558b0484b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 -- 172.21.15.32:0/1536233826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04840f90 msgr2=0x558b0484b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04840f90 0x558b0484b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 -- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04842b20 msgr2=0x558b04852990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2069d9b640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04842b20 0x558b04852990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04842b20 0x558b04852990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 -- 172.21.15.32:0/1536233826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b0457e6b0 con 0x558b04842160 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f206a59c640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f205c0039d0 tx=0x7f205c003a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4837d0f771cbb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c029060 con 0x558b04842160 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f205c003fa0 con 0x558b04842160 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f205c0035f0 con 0x558b04842160 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f205c0041a0 con 0x558b04842160 2024-04-23T18:14:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2073831740 10 client.15102 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b04855d00 con 0x558b04842160 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.828+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c0035f0 con 0x558b04842160 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f205c004750 con 0x558b04842160 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 10 client.15102 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 10 client.15102 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 20 client.15102 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b04857fa0 0x558b04859410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 4 client.15102 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b047af670 con 0x558b04857fa0 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2069d9b640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b04857fa0 0x558b04859410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2069d9b640 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b04857fa0 0x558b04859410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=251dd09cfc5f4dc1 server_cookie=dd479264969d73aa in_seq=0 out_seq=0 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.829+0000 7f2051ffb640 10 client.15102 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2051ffb640 1 -- 172.21.15.32:0/1536233826 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b047af670 con 0x558b04857fa0 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2051ffb640 10 client.15102 handle_command_reply: tid=0 2024-04-23T18:14:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 client.15102 shutdown 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 msgr2=0x558b0484b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f205c0039d0 tx=0x7f205c003a00 comp rx=0 tx=0).stop 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 shutdown_connections 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b04857fa0 0x558b04859410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04842b20 0x558b04852990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2051ffb640 0 client.15102 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b0484b950 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 --2- 172.21.15.32:0/1536233826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04840f90 0x558b0484b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 >> 172.21.15.32:0/1536233826 conn(0x558b04838070 msgr2=0x558b048371d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 shutdown_connections 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 1 -- 172.21.15.32:0/1536233826 wait complete. 2024-04-23T18:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.830+0000 7f2073831740 20 client.15102 trim_cache size 0 max 16384 2024-04-23T18:14:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 Processor -- start 2024-04-23T18:14:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 -- start start 2024-04-23T18:14:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b04857160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b0484b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b04857750 con 0x558b04842160 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b04857e20 con 0x558b04840f90 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2073831740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558b048584f0 con 0x558b04842b20 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35658/0 (socket says 172.21.15.32:35658) 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 -- 172.21.15.32:0/1508053062 learned_addr learned my addr 172.21.15.32:0/1508053062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2069d9b640 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b04857160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 -- 172.21.15.32:0/1508053062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 msgr2=0x558b0484b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b0484b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 -- 172.21.15.32:0/1508053062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 msgr2=0x558b04857160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b04857160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 -- 172.21.15.32:0/1508053062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b04841be0 con 0x558b04842160 2024-04-23T18:14:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f206a59c640 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f205c004d60 tx=0x7f205c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d868432ce886e07a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2053fff640 1 -- 172.21.15.32:0/1508053062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c034070 con 0x558b04842160 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2053fff640 1 -- 172.21.15.32:0/1508053062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c011440 con 0x558b04842160 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.849+0000 7f2053fff640 1 -- 172.21.15.32:0/1508053062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c029070 con 0x558b04842160 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- 172.21.15.32:0/1508053062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 msgr2=0x558b04849280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f205c004d60 tx=0x7f205c00b8e0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- 172.21.15.32:0/1508053062 shutdown_connections 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842b20 0x558b0484b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842160 0x558b04849280 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- 172.21.15.32:0/1508053062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b04857160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- 172.21.15.32:0/1508053062 >> 172.21.15.32:0/1508053062 conn(0x558b04838070 msgr2=0x558b048330b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- 172.21.15.32:0/1508053062 shutdown_connections 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- 172.21.15.32:0/1508053062 wait complete. 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- start start 2024-04-23T18:14:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 Processor -- start 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b0484d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 0x558b0484d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b04859290 con 0x558b04842b20 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b04858bc0 con 0x558b04840f90 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.850+0000 7f2073831740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558b04859960 con 0x558b04842160 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2073831740 10 client.0 _ll_register_callbacks cb 0x7f205c00bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35664/0 (socket says 172.21.15.32:35664) 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 -- 172.21.15.32:0/2692861665 learned_addr learned my addr 172.21.15.32:0/2692861665 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f206a59c640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 0x558b0484d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 -- 172.21.15.32:0/2692861665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 msgr2=0x558b0484d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 0x558b0484d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 -- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 msgr2=0x558b0484d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b0484d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 -- 172.21.15.32:0/2692861665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b04850be0 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f206a59c640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 0x558b0484d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2069d9b640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2060009850 tx=0x7f20600318a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa72fd261cca5b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206004d020 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2060049570 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f205c011440 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2060046040 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2073831740 10 client.15108 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.851+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b04851ea0 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c011440 con 0x558b04842b20 2024-04-23T18:14:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2060049050 con 0x558b04842b20 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 10 client.15108 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 10 client.15108 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 20 client.15108 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b048c2680 0x558b048beac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 4 client.15108 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "status"}) -- 0x558b047a94e0 con 0x558b048c2680 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f206a59c640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b048c2680 0x558b048beac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f2051ffb640 10 client.15108 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.852+0000 7f206a59c640 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b048c2680 0x558b048beac0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29c9491294944502 server_cookie=f90388c5ca06accc in_seq=0 out_seq=0 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2051ffb640 1 -- 172.21.15.32:0/2692861665 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558b047a94e0 con 0x558b048c2680 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2051ffb640 10 client.15108 handle_command_reply: tid=0 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 client.15108 shutdown 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 msgr2=0x558b048505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2060009850 tx=0x7f20600318a0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 shutdown_connections 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558b048c2680 0x558b048beac0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b04842b20 0x558b048505a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b04842160 0x558b0484d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2051ffb640 0 client.15108 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 --2- 172.21.15.32:0/2692861665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b04840f90 0x558b0484d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 >> 172.21.15.32:0/2692861665 conn(0x558b04838070 msgr2=0x558b04849110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 shutdown_connections 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 1 -- 172.21.15.32:0/2692861665 wait complete. 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2073831740 20 client.15108 trim_cache size 0 max 16384 2024-04-23T18:14:33.853 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24551, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 87.847251584999995, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 93.000935104999996 2024-04-23T18:14:33.854 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:33.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 msgr2=0x7f20540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 0x7f20540a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f20640ce3c0 tx=0x7f2064066fe0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 msgr2=0x7f206c19f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f206400eaa0 tx=0x7f206400ef60 comp rx=0 tx=0).stop 2024-04-23T18:14:33.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 shutdown_connections 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f206c107e20 0x7f206c19f850 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f20540a3bb0 0x7f20540a6070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c1054f0 0x7f206c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 --2- 172.21.15.32:0/952589459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f206c102bc0 0x7f206c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.853+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 >> 172.21.15.32:0/952589459 conn(0x7f206c0fc5f0 msgr2=0x7f206c103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.854+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 shutdown_connections 2024-04-23T18:14:33.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:33.854+0000 7f2070b22640 1 -- 172.21.15.32:0/952589459 wait complete. 2024-04-23T18:14:33.858 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T18:14:33.859 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:14:33.868 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:14:33.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-23T18:14:34.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:33 smithi032 ceph-mon[29221]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:34.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:33 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4161130975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:33 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2450159227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.114 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:34.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:33 smithi064 ceph-mon[33062]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:34.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:33 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4161130975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:33 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2450159227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:33 smithi040 ceph-mon[32626]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:34.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:33 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4161130975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:33 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2450159227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.026+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2634543077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64102ac0 msgr2=0x7f7c64102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.026+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2634543077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64102ac0 0x7f7c64102ee0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7c580099c0 tx=0x7f7c58009cd0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.026+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2634543077 shutdown_connections 2024-04-23T18:14:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2634543077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64104660 0x7f7c6410af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2634543077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64102ac0 0x7f7c64102ee0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2634543077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 0x7f7c64104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2634543077 >> 172.21.15.32:0/2634543077 conn(0x7f7c640fe060 msgr2=0x7f7c641004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2634543077 shutdown_connections 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2634543077 wait complete. 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 Processor -- start 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- start start 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 0x7f7c640767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 0x7f7c64074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6410e530 con 0x7f7c64104660 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6410e3b0 con 0x7f7c64102ac0 2024-04-23T18:14:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.027+0000 7f7c6b07f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6410e6b0 con 0x7f7c64103cc0 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c68e1b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 0x7f7c640767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35682/0 (socket says 172.21.15.32:35682) 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 -- 172.21.15.32:0/2671559817 learned_addr learned my addr 172.21.15.32:0/2671559817 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c63fff700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 0x7f7c64074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 -- 172.21.15.32:0/2671559817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 msgr2=0x7f7c64074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 0x7f7c64074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 -- 172.21.15.32:0/2671559817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 msgr2=0x7f7c640767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 0x7f7c640767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 -- 172.21.15.32:0/2671559817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c64075c20 con 0x7f7c64104660 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c68e1b700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 0x7f7c640767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6961c700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00ef30 tx=0x7f7c5c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0092e0 con 0x7f7c64104660 2024-04-23T18:14:36.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c641a2d30 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c5c004bb0 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.028+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c004020 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.029+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c641a31e0 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.029+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c64069730 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.031+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7c5c01b030 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.031+0000 7f7c61ffb700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 0x7f7c540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.031+0000 7f7c68e1b700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 0x7f7c540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.031+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7c5c0cfe70 con 0x7f7c64104660 2024-04-23T18:14:36.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.032+0000 7f7c68e1b700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 0x7f7c540a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b580 tx=0x7f7c58005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:36.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.036+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c5c022020 con 0x7f7c64104660 2024-04-23T18:14:36.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:35 smithi064 ceph-mon[33062]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:14:36.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.261+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f7c64062e80 con 0x7f7c540a3c40 2024-04-23T18:14:36.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:35 smithi032 ceph-mon[29221]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:14:36.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.262+0000 7f7c61ffb700 1 -- 172.21.15.32:0/2671559817 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f7c64062e80 con 0x7f7c540a3c40 2024-04-23T18:14:36.256 INFO:teuthology.orchestra.run.smithi032.stdout:Backend: cephadm 2024-04-23T18:14:36.256 INFO:teuthology.orchestra.run.smithi032.stdout:Available: Yes 2024-04-23T18:14:36.256 INFO:teuthology.orchestra.run.smithi032.stdout:Paused: No 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 msgr2=0x7f7c540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 0x7f7c540a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b580 tx=0x7f7c58005fb0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 msgr2=0x7f7c64075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00ef30 tx=0x7f7c5c00c3a0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 shutdown_connections 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c64104660 0x7f7c64075370 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f7c540a3c40 0x7f7c540a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c64102ac0 0x7f7c640767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 --2- 172.21.15.32:0/2671559817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c64103cc0 0x7f7c64074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.264+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 >> 172.21.15.32:0/2671559817 conn(0x7f7c640fe060 msgr2=0x7f7c641070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:36.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.265+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 shutdown_connections 2024-04-23T18:14:36.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:36.265+0000 7f7c6b07f700 1 -- 172.21.15.32:0/2671559817 wait complete. 2024-04-23T18:14:36.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:35 smithi040 ceph-mon[32626]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:14:36.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-23T18:14:37.249 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:38.169 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:37 smithi040 ceph-mon[32626]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:38.169 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:37 smithi040 ceph-mon[32626]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:38.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:37 smithi064 ceph-mon[33062]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:38.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:37 smithi064 ceph-mon[33062]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:38.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:37 smithi032 ceph-mon[29221]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:14:38.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:37 smithi032 ceph-mon[29221]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 -- 172.21.15.32:0/2960489620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc103540 msgr2=0x7f97cc109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/2960489620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc103540 0x7f97cc109df0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b0d0 tx=0x7f97c400b3e0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 -- 172.21.15.32:0/2960489620 shutdown_connections 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/2960489620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc103540 0x7f97cc109df0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/2960489620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 0x7f97cc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/2960489620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc102ba0 0x7f97cc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 -- 172.21.15.32:0/2960489620 >> 172.21.15.32:0/2960489620 conn(0x7f97cc0fcf40 msgr2=0x7f97cc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 -- 172.21.15.32:0/2960489620 shutdown_connections 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.107+0000 7f97d2a9b700 1 -- 172.21.15.32:0/2960489620 wait complete. 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 Processor -- start 2024-04-23T18:14:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 -- start start 2024-04-23T18:14:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 0x7f97cc198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc103540 0x7f97cc1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d420 con 0x7f97cc102ba0 2024-04-23T18:14:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d2a0 con 0x7f97cc103540 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d2a9b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d5a0 con 0x7f97cc1019a0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37230/0 (socket says 172.21.15.32:37230) 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97cbfff700 1 -- 172.21.15.32:0/4001820631 learned_addr learned my addr 172.21.15.32:0/4001820631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d0837700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 0x7f97cc198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.108+0000 7f97d1038700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc103540 0x7f97cc1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 -- 172.21.15.32:0/4001820631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 msgr2=0x7f97cc198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 0x7f97cc198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 -- 172.21.15.32:0/4001820631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc103540 msgr2=0x7f97cc1a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc103540 0x7f97cc1a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 -- 172.21.15.32:0/4001820631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc1a0700 con 0x7f97cc102ba0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97cbfff700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c9e0 tx=0x7f97b800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8007ac0 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b8007c60 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1a09f0 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8018660 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.109+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc1a0f30 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.110+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97cc069730 con 0x7f97cc102ba0 2024-04-23T18:14:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.112+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f97b800f450 con 0x7f97cc102ba0 2024-04-23T18:14:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.113+0000 7f97c9ffb700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 0x7f97bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.113+0000 7f97d0837700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 0x7f97bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.113+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f97b80d0580 con 0x7f97cc102ba0 2024-04-23T18:14:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.113+0000 7f97d0837700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 0x7f97bc0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97cc102a00 tx=0x7f97c000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:39.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.117+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97b8014030 con 0x7f97cc102ba0 2024-04-23T18:14:39.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.335+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f97cc0ff040 con 0x7f97bc0a3c40 2024-04-23T18:14:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.350+0000 7f97c9ffb700 1 -- 172.21.15.32:0/4001820631 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f97cc0ff040 con 0x7f97bc0a3c40 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.a smithi032 running (102s) 16s ago 101s 14.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 519a8222111b 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.b smithi040 running (98s) 32s ago 98s 28.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d8be8c1e1b90 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.c smithi064 running (113s) 32s ago 113s 20.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b5a8e1bcf9a0 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.d smithi032 running (2m) 16s ago 2m 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2c566e5388ca 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.e smithi040 running (119s) 32s ago 119s 19.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 0cd2f5bd6c02 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.f smithi064 running (116s) 32s ago 116s 19.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 42a2f0b8869d 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.g smithi032 running (105s) 16s ago 104s 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 913fe3a9127d 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.h smithi040 running (110s) 32s ago 110s 18.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e24e944140a4 2024-04-23T18:14:39.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.i smithi064 running (107s) 32s ago 107s 17.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6db7a1fe6ce2 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mds.j smithi032 running (2m) 16s ago 2m 16.5M - 19.0.0-3156-gd69f13d8 da4b004c09e8 3df7c5d86506 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mds.k smithi040 running (2m) 32s ago 2m 14.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6a550abe154a 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mds.l smithi064 running (2m) 32s ago 2m 16.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 02798a04ebca 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.x smithi032 *:9283,8765 running (9m) 16s ago 9m 514M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d1cd4cf4d2da 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.y smithi040 *:8443,8765 running (7m) 32s ago 7m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 24faf3f31405 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.z smithi064 *:8443,8765 running (7m) 32s ago 7m 452M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c5694f5f7bd5 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a smithi032 running (9m) 16s ago 9m 49.1M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 1f840779cda8 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b smithi040 running (7m) 32s ago 7m 38.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 34f7777fe018 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c smithi064 running (7m) 32s ago 7m 39.3M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 70b853e25ec9 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 16s ago 7m 55.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 bffc293312df 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi040 running (6m) 32s ago 6m 46.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 02eff54c2ffd 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi064 running (6m) 32s ago 6m 61.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 da40e7fee159 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (6m) 16s ago 6m 44.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 7fe44ddf65b1 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi040 running (5m) 32s ago 5m 55.7M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 8a96acc422d8 2024-04-23T18:14:39.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi064 running (5m) 32s ago 5m 54.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 64ca288e8d9c 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi032 running (4m) 16s ago 4m 50.3M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 6bd37cd0064c 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi040 running (4m) 32s ago 4m 54.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 18225a6d3dd4 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 smithi064 running (3m) 32s ago 3m 55.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 cd4bb9ac4b6a 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9 smithi032 running (3m) 16s ago 3m 41.7M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 8d42e77eded0 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 smithi040 running (2m) 32s ago 2m 42.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 a2c255e16d55 2024-04-23T18:14:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 smithi064 running (2m) 32s ago 2m 52.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 9c7cf827def9 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.352+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 msgr2=0x7f97bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.352+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 0x7f97bc0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97cc102a00 tx=0x7f97c000a3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.352+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 msgr2=0x7f97cc199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.352+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c9e0 tx=0x7f97b800ccf0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 shutdown_connections 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f97bc0a3c40 0x7f97bc0a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97cc103540 0x7f97cc1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97cc1019a0 0x7f97cc198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 --2- 172.21.15.32:0/4001820631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97cc102ba0 0x7f97cc199030 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 >> 172.21.15.32:0/4001820631 conn(0x7f97cc0fcf40 msgr2=0x7f97cc0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 shutdown_connections 2024-04-23T18:14:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:39.353+0000 7f97d2a9b700 1 -- 172.21.15.32:0/4001820631 wait complete. 2024-04-23T18:14:40.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:39 smithi032 ceph-mon[29221]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:40.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-23T18:14:40.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:39 smithi064 ceph-mon[33062]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:40.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:39 smithi040 ceph-mon[32626]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:14:41.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:40 smithi032 ceph-mon[29221]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:41.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:40 smithi064 ceph-mon[33062]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:41.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:40 smithi040 ceph-mon[32626]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:42.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:41 smithi032 ceph-mon[29221]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:14:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 -- 172.21.15.32:0/2738993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac103cc0 msgr2=0x7f11ac104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 --2- 172.21.15.32:0/2738993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac103cc0 0x7f11ac104120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f11a00099c0 tx=0x7f11a0009cd0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 -- 172.21.15.32:0/2738993153 shutdown_connections 2024-04-23T18:14:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 --2- 172.21.15.32:0/2738993153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac104660 0x7f11ac10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 --2- 172.21.15.32:0/2738993153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac102ac0 0x7f11ac102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 --2- 172.21.15.32:0/2738993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac103cc0 0x7f11ac104120 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 -- 172.21.15.32:0/2738993153 >> 172.21.15.32:0/2738993153 conn(0x7f11ac0fe060 msgr2=0x7f11ac1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 -- 172.21.15.32:0/2738993153 shutdown_connections 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.122+0000 7f11b4747700 1 -- 172.21.15.32:0/2738993153 wait complete. 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 Processor -- start 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 -- start start 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac103cc0 0x7f11ac199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 0x7f11ac1a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11ac10e5f0 con 0x7f11ac102ac0 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f11ac10e470 con 0x7f11ac104660 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b4747700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f11ac10e770 con 0x7f11ac103cc0 2024-04-23T18:14:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37244/0 (socket says 172.21.15.32:37244) 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 -- 172.21.15.32:0/2805691088 learned_addr learned my addr 172.21.15.32:0/2805691088 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b1ce2700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac103cc0 0x7f11ac199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b2ce4700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 0x7f11ac1a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 -- 172.21.15.32:0/2805691088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac103cc0 msgr2=0x7f11ac199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac103cc0 0x7f11ac199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 -- 172.21.15.32:0/2805691088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 msgr2=0x7f11ac1a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 0x7f11ac1a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 -- 172.21.15.32:0/2805691088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ac1a08a0 con 0x7f11ac102ac0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b2ce4700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 0x7f11ac1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.123+0000 7f11b24e3700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f11a400b500 tx=0x7f11a400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.124+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4015020 con 0x7f11ac102ac0 2024-04-23T18:14:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.124+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11a4011ce0 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.124+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4010d20 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.124+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ac1a0b90 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.124+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ac1a10d0 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.125+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ac069730 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.126+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f11a4018020 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.127+0000 7f11ab7fe700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.127+0000 7f11b1ce2700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.127+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f11a40cfb20 con 0x7f11ac102ac0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.127+0000 7f11b1ce2700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 0x7f11940a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009990 tx=0x7f11a000b870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.132+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f11a401f020 con 0x7f11ac102ac0 2024-04-23T18:14:42.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.349+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f11ac062e80 con 0x7f11940a3c40 2024-04-23T18:14:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.355+0000 7f11ab7fe700 1 -- 172.21.15.32:0/2805691088 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f11ac062e80 con 0x7f11940a3c40 2024-04-23T18:14:42.349 INFO:teuthology.orchestra.run.smithi032.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-23T18:14:42.349 INFO:teuthology.orchestra.run.smithi032.stdout:mds.all 12/12 35s ago 2m smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi064=c;smithi064=f;smithi064=i;smithi064=l;count:12 2024-04-23T18:14:42.349 INFO:teuthology.orchestra.run.smithi032.stdout:mgr 3/3 35s ago 7m smithi032=x;smithi040=y;smithi064=z;count:3 2024-04-23T18:14:42.349 INFO:teuthology.orchestra.run.smithi032.stdout:mon 3/3 35s ago 8m smithi032:172.21.15.32=a;smithi040:172.21.15.40=b;smithi064:172.21.15.64=c;count:3 2024-04-23T18:14:42.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd 12 35s ago - 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 msgr2=0x7f11940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 0x7f11940a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009990 tx=0x7f11a000b870 comp rx=0 tx=0).stop 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 msgr2=0x7f11ac198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f11a400b500 tx=0x7f11a400b810 comp rx=0 tx=0).stop 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 shutdown_connections 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11ac104660 0x7f11ac1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ac102ac0 0x7f11ac198c50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 --2- 172.21.15.32:0/2805691088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11ac103cc0 0x7f11ac199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 >> 172.21.15.32:0/2805691088 conn(0x7f11ac0fe060 msgr2=0x7f11ac106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 shutdown_connections 2024-04-23T18:14:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:42.358+0000 7f11b4747700 1 -- 172.21.15.32:0/2805691088 wait complete. 2024-04-23T18:14:42.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:41 smithi040 ceph-mon[32626]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:14:42.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:41 smithi064 ceph-mon[33062]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:14:43.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-23T18:14:43.421 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:44.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:43 smithi032 ceph-mon[29221]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-23T18:14:44.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:43 smithi032 ceph-mon[29221]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:44.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:43 smithi040 ceph-mon[32626]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-23T18:14:44.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:43 smithi040 ceph-mon[32626]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:44.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:43 smithi064 ceph-mon[33062]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-23T18:14:44.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:43 smithi064 ceph-mon[33062]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:45.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 -- 172.21.15.32:0/2723818157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0101340 msgr2=0x7fc2d0103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 --2- 172.21.15.32:0/2723818157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0101340 0x7fc2d0103770 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0009990 tx=0x7fc2c0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 -- 172.21.15.32:0/2723818157 shutdown_connections 2024-04-23T18:14:45.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 --2- 172.21.15.32:0/2723818157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 --2- 172.21.15.32:0/2723818157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0101340 0x7fc2d0103770 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 --2- 172.21.15.32:0/2723818157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0103cb0 0x7fc2d0106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 -- 172.21.15.32:0/2723818157 >> 172.21.15.32:0/2723818157 conn(0x7fc2d00fcf60 msgr2=0x7fc2d00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 -- 172.21.15.32:0/2723818157 shutdown_connections 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.073+0000 7fc2d6653700 1 -- 172.21.15.32:0/2723818157 wait complete. 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 Processor -- start 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 -- start start 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0101340 0x7fc2d019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d01a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0114c00 con 0x7fc2d0103cb0 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0114a80 con 0x7fc2d0106680 2024-04-23T18:14:45.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2d6653700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0114d80 con 0x7fc2d0101340 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37266/0 (socket says 172.21.15.32:37266) 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.074+0000 7fc2c7fff700 1 -- 172.21.15.32:0/162830860 learned_addr learned my addr 172.21.15.32:0/162830860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2cffff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0101340 0x7fc2d019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2d4bf0700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d01a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 -- 172.21.15.32:0/162830860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0101340 msgr2=0x7fc2d019f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0101340 0x7fc2d019f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 -- 172.21.15.32:0/162830860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 msgr2=0x7fc2d01a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d01a68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 -- 172.21.15.32:0/162830860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d01a6f50 con 0x7fc2d0103cb0 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2d4bf0700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d01a68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2c7fff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800b560 tx=0x7fc2b800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8015020 con 0x7fc2d0103cb0 2024-04-23T18:14:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d01a7240 con 0x7fc2d0103cb0 2024-04-23T18:14:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b8011ce0 con 0x7fc2d0103cb0 2024-04-23T18:14:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8010d20 con 0x7fc2d0103cb0 2024-04-23T18:14:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.075+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d01a7850 con 0x7fc2d0103cb0 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.076+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d0069730 con 0x7fc2d0103cb0 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.078+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc2b8018020 con 0x7fc2d0103cb0 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.078+0000 7fc2cdffb700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 0x7fc2bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.078+0000 7fc2cffff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 0x7fc2bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.078+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc2b80d0120 con 0x7fc2d0103cb0 2024-04-23T18:14:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.079+0000 7fc2cffff700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 0x7fc2bc0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000b5c0 tx=0x7fc2c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.082+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2b801f020 con 0x7fc2d0103cb0 2024-04-23T18:14:45.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.295+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc2d0062e80 con 0x7fc2bc0a3c40 2024-04-23T18:14:45.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.297+0000 7fc2cdffb700 1 -- 172.21.15.32:0/162830860 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7fc2d0062e80 con 0x7fc2bc0a3c40 2024-04-23T18:14:45.290 INFO:teuthology.orchestra.run.smithi032.stdout:HOST ADDR LABELS STATUS 2024-04-23T18:14:45.291 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 172.21.15.32 2024-04-23T18:14:45.291 INFO:teuthology.orchestra.run.smithi032.stdout:smithi040 172.21.15.40 2024-04-23T18:14:45.291 INFO:teuthology.orchestra.run.smithi032.stdout:smithi064 172.21.15.64 2024-04-23T18:14:45.291 INFO:teuthology.orchestra.run.smithi032.stdout:3 hosts in cluster 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 msgr2=0x7fc2bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 0x7fc2bc0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000b5c0 tx=0x7fc2c0005fb0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 msgr2=0x7fc2d019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800b560 tx=0x7fc2b800b870 comp rx=0 tx=0).stop 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 shutdown_connections 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fc2bc0a3c40 0x7fc2bc0a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d0106680 0x7fc2d01a68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc2d0101340 0x7fc2d019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 --2- 172.21.15.32:0/162830860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d0103cb0 0x7fc2d019f830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 >> 172.21.15.32:0/162830860 conn(0x7fc2d00fcf60 msgr2=0x7fc2d0104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 shutdown_connections 2024-04-23T18:14:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:45.299+0000 7fc2d6653700 1 -- 172.21.15.32:0/162830860 wait complete. 2024-04-23T18:14:46.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-23T18:14:46.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:45 smithi032 ceph-mon[29221]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:46.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:46.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:45 smithi040 ceph-mon[32626]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:46.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:45 smithi064 ceph-mon[33062]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:47.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:46 smithi032 ceph-mon[29221]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:47.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:46 smithi040 ceph-mon[32626]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:47.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:46 smithi064 ceph-mon[33062]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:48.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- 172.21.15.32:0/3864371596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d18103560 msgr2=0x7f6d18109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:48.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/3864371596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d18103560 0x7f6d18109e10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000b0d0 tx=0x7f6d1000b3e0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- 172.21.15.32:0/3864371596 shutdown_connections 2024-04-23T18:14:48.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/3864371596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d18103560 0x7f6d18109e10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/3864371596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d181019c0 0x7f6d18101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/3864371596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 0x7f6d18103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- 172.21.15.32:0/3864371596 >> 172.21.15.32:0/3864371596 conn(0x7f6d180fcf60 msgr2=0x7f6d180ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- 172.21.15.32:0/3864371596 shutdown_connections 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- 172.21.15.32:0/3864371596 wait complete. 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 Processor -- start 2024-04-23T18:14:48.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.096+0000 7f6d1eb32700 1 -- start start 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 0x7f6d18197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d18103560 0x7f6d1819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810d460 con 0x7f6d181019c0 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810d2e0 con 0x7f6d18102bc0 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1eb32700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810d5e0 con 0x7f6d18103560 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58356/0 (socket says 172.21.15.32:58356) 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 -- 172.21.15.32:0/737194304 learned_addr learned my addr 172.21.15.32:0/737194304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:48.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d17fff700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 0x7f6d18197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1d0cf700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d18103560 0x7f6d1819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 -- 172.21.15.32:0/737194304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d18103560 msgr2=0x7f6d1819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d18103560 0x7f6d1819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 -- 172.21.15.32:0/737194304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 msgr2=0x7f6d18197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 0x7f6d18197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 -- 172.21.15.32:0/737194304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1819f6b0 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.097+0000 7f6d1c8ce700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00be70 tx=0x7f6d0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.098+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c016070 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.098+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d0c0193b0 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.098+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1806b1d0 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.098+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c009580 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.098+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1806b710 con 0x7f6d181019c0 2024-04-23T18:14:48.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.099+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d18069730 con 0x7f6d181019c0 2024-04-23T18:14:48.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.101+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6d0c019560 con 0x7f6d181019c0 2024-04-23T18:14:48.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.101+0000 7f6d15ffb700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 0x7f6d080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:48.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.101+0000 7f6d17fff700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 0x7f6d080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:48.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.101+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6d0c0d1ef0 con 0x7f6d181019c0 2024-04-23T18:14:48.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.102+0000 7f6d17fff700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 0x7f6d080a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005950 tx=0x7f6d0400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:48.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.105+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d0c09e900 con 0x7f6d181019c0 2024-04-23T18:14:48.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:47 smithi032 ceph-mon[29221]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:48.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.322+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f6d1806ba20 con 0x7f6d080a3cc0 2024-04-23T18:14:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.324+0000 7f6d15ffb700 1 -- 172.21.15.32:0/737194304 <== mgr.14152 v2:172.21.15.32:6800/3652066649 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f6d1806ba20 con 0x7f6d080a3cc0 2024-04-23T18:14:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-23T18:14:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-23T18:14:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:14:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:14:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 msgr2=0x7f6d080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 0x7f6d080a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005950 tx=0x7f6d0400a300 comp rx=0 tx=0).stop 2024-04-23T18:14:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 msgr2=0x7f6d18197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00be70 tx=0x7f6d0c00dc90 comp rx=0 tx=0).stop 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 shutdown_connections 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d18103560 0x7f6d1819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d181019c0 0x7f6d18197a50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6d080a3cc0 0x7f6d080a6180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 --2- 172.21.15.32:0/737194304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102bc0 0x7f6d18197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 >> 172.21.15.32:0/737194304 conn(0x7f6d180fcf60 msgr2=0x7f6d18105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 shutdown_connections 2024-04-23T18:14:48.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:48.326+0000 7f6d1eb32700 1 -- 172.21.15.32:0/737194304 wait complete. 2024-04-23T18:14:48.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:47 smithi040 ceph-mon[32626]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:48.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:47 smithi064 ceph-mon[33062]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:49.107 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:14:49.117 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:14:49.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T18:14:49.366 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:50.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:49 smithi032 ceph-mon[29221]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:50.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:49 smithi032 ceph-mon[29221]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:50.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:49 smithi040 ceph-mon[32626]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:50.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:49 smithi040 ceph-mon[32626]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:50.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:49 smithi064 ceph-mon[33062]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:14:50.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:49 smithi064 ceph-mon[33062]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/1793370491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 msgr2=0x7f2fb01082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/1793370491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb01082a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009990 tx=0x7f2f98009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/1793370491 shutdown_connections 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/1793370491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb01087e0 0x7f2fb010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/1793370491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0103580 0x7f2fb0105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/1793370491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb01082a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/1793370491 >> 172.21.15.32:0/1793370491 conn(0x7f2fb00fcf30 msgr2=0x7f2fb00ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/1793370491 shutdown_connections 2024-04-23T18:14:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.197+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/1793370491 wait complete. 2024-04-23T18:14:51.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.198+0000 7f2fb5c3f700 1 Processor -- start 2024-04-23T18:14:51.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.198+0000 7f2fb5c3f700 1 -- start start 2024-04-23T18:14:51.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb0103580 0x7f2fb0102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb01087e0 0x7f2fb0101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010e590 con 0x7f2fb0105eb0 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010e410 con 0x7f2fb01087e0 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fb5c3f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010e710 con 0x7f2fb0103580 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58374/0 (socket says 172.21.15.32:58374) 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 -- 172.21.15.32:0/763347536 learned_addr learned my addr 172.21.15.32:0/763347536 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2faf7fe700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb0103580 0x7f2fb0102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2faffff700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb01087e0 0x7f2fb0101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 -- 172.21.15.32:0/763347536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb0103580 msgr2=0x7f2fb0102ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb0103580 0x7f2fb0102ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 -- 172.21.15.32:0/763347536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb01087e0 msgr2=0x7f2fb0101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb01087e0 0x7f2fb0101a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 -- 172.21.15.32:0/763347536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb0102320 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fa6ffd700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2f980036f0 tx=0x7f2f980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.199+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98027070 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.200+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb01a2ce0 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.200+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f98022470 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.200+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9800f6c0 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.200+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb01a3130 con 0x7f2fb0105eb0 2024-04-23T18:14:51.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.201+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb002df80 con 0x7f2fb0105eb0 2024-04-23T18:14:51.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.202+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2f9802a070 con 0x7f2fb0105eb0 2024-04-23T18:14:51.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.203+0000 7f2fad7fa700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 0x7f2f9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:51.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.203+0000 7f2faf7fe700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 0x7f2f9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:51.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.203+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2f980d16c0 con 0x7f2fb0105eb0 2024-04-23T18:14:51.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.203+0000 7f2faf7fe700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 0x7f2f9c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000ac50 tx=0x7f2fa000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:51.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.206+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f98013070 con 0x7f2fb0105eb0 2024-04-23T18:14:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.462+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2fb002e350 con 0x7f2fb0105eb0 2024-04-23T18:14:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.462+0000 7f2fad7fa700 1 -- 172.21.15.32:0/763347536 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3036 (secure 0 0 0) 0x7f2f9809e160 con 0x7f2fb0105eb0 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:e21 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 21 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T18:14:51.457 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-23T18:13:05.375853+0000 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-23T18:13:10.043413+0000 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 300 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 0 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 3 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1,2 2024-04-23T18:14:51.458 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=24551,1=24515,2=24499} 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 0 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:qdb_cluster leader: 24551 members: 24499,24515,24551 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b{0:24551} state up:active seq 3 addr [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c{0:24487} state up:standby-replay seq 1 addr [v2:172.21.15.64:6836/1625538187,v1:172.21.15.64:6837/1625538187] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.h{1:24515} state up:active seq 6 addr [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e{1:24479} state up:standby-replay seq 1 addr [v2:172.21.15.40:6834/1068324383,v1:172.21.15.40:6835/1068324383] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.i{2:24499} state up:active seq 6 addr [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f{2:24475} state up:standby-replay seq 1 addr [v2:172.21.15.64:6834/669117639,v1:172.21.15.64:6835/669117639] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.j{-1:14553} state up:standby seq 1 addr [v2:172.21.15.32:6834/1839255789,v1:172.21.15.32:6835/1839255789] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.32:6836/2912347592,v1:172.21.15.32:6837/2912347592] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.g{-1:14631} state up:standby seq 1 addr [v2:172.21.15.32:6838/156606909,v1:172.21.15.32:6839/156606909] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.32:6840/2029849926,v1:172.21.15.32:6841/2029849926] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.64:6832/2951030033,v1:172.21.15.64:6833/2951030033] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.k{-1:24449} state up:standby seq 1 addr [v2:172.21.15.40:6832/768829407,v1:172.21.15.40:6833/768829407] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 msgr2=0x7f2f9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 0x7f2f9c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000ac50 tx=0x7f2fa000a380 comp rx=0 tx=0).stop 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 msgr2=0x7f2fb0101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2f980036f0 tx=0x7f2f980047b0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 shutdown_connections 2024-04-23T18:14:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f2f9c0a3c40 0x7f2f9c0a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb01087e0 0x7f2fb0101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fb0103580 0x7f2fb0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 --2- 172.21.15.32:0/763347536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fb0105eb0 0x7f2fb0101530 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 >> 172.21.15.32:0/763347536 conn(0x7f2fb00fcf30 msgr2=0x7f2fb0103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 shutdown_connections 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:51.465+0000 7f2fb5c3f700 1 -- 172.21.15.32:0/763347536 wait complete. 2024-04-23T18:14:51.462 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:52.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:51 smithi032 ceph-mon[29221]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:52.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:51 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/763347536' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:14:52.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-23T18:14:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:51 smithi040 ceph-mon[32626]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:52.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:51 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/763347536' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:14:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:52.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:51 smithi064 ceph-mon[33062]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:14:52.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:51 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/763347536' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.169+0000 7f0bf0269700 1 -- 172.21.15.32:0/58630002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 msgr2=0x7f0be8104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.169+0000 7f0bf0269700 1 --2- 172.21.15.32:0/58630002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8104120 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80099c0 tx=0x7f0bd8009cd0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- 172.21.15.32:0/58630002 shutdown_connections 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- 172.21.15.32:0/58630002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8104660 0x7f0be810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- 172.21.15.32:0/58630002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8102ac0 0x7f0be8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- 172.21.15.32:0/58630002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8104120 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- 172.21.15.32:0/58630002 >> 172.21.15.32:0/58630002 conn(0x7f0be80fe060 msgr2=0x7f0be81004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- 172.21.15.32:0/58630002 shutdown_connections 2024-04-23T18:14:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- 172.21.15.32:0/58630002 wait complete. 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 Processor -- start 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- start start 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8102ac0 0x7f0be8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 0x7f0be81a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810e560 con 0x7f0be8103cc0 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810e3e0 con 0x7f0be8104660 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.170+0000 7f0bf0269700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810e6e0 con 0x7f0be8102ac0 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58380/0 (socket says 172.21.15.32:58380) 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 -- 172.21.15.32:0/610142906 learned_addr learned my addr 172.21.15.32:0/610142906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bee005700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8102ac0 0x7f0be8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bee806700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 0x7f0be81a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 -- 172.21.15.32:0/610142906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8102ac0 msgr2=0x7f0be8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8102ac0 0x7f0be8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 -- 172.21.15.32:0/610142906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 msgr2=0x7f0be81a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 0x7f0be81a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 -- 172.21.15.32:0/610142906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be81a0830 con 0x7f0be8103cc0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bee806700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 0x7f0be81a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bed804700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b580 tx=0x7f0bd800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8027070 con 0x7f0be8103cc0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be81a0ac0 con 0x7f0be8103cc0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bd8022920 con 0x7f0be8103cc0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8009e60 con 0x7f0be8103cc0 2024-04-23T18:14:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.171+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be81a0f70 con 0x7f0be8103cc0 2024-04-23T18:14:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.172+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be8069730 con 0x7f0be8103cc0 2024-04-23T18:14:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.173+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0bd802a070 con 0x7f0be8103cc0 2024-04-23T18:14:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.174+0000 7f0bdf7fe700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.174+0000 7f0bee005700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.174+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0bd80d1120 con 0x7f0be8103cc0 2024-04-23T18:14:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.175+0000 7f0bee005700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 0x7f0bd40a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0be8103b20 tx=0x7f0be4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.179+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bd8013070 con 0x7f0be8103cc0 2024-04-23T18:14:54.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:53 smithi032 ceph-mon[29221]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:54.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:53 smithi040 ceph-mon[32626]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:54.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.390+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0be8063650 con 0x7f0be8103cc0 2024-04-23T18:14:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.391+0000 7f0bdf7fe700 1 -- 172.21.15.32:0/610142906 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5233 (secure 0 0 0) 0x7f0bd809dce0 con 0x7f0be8103cc0 2024-04-23T18:14:54.385 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 88 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:fsid cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-23T18:04:51.553089+0000 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-23T18:13:57.185219+0000 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:crush_version 26 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:full_ratio 0.95 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:backfillfull_ratio 0.9 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:nearfull_ratio 0.85 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:require_min_compat_client luminous 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:min_compat_client luminous 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:require_osd_release squid 2024-04-23T18:14:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:stretch_mode_enabled false 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:max_osd 12 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6802/1995005420,v1:172.21.15.32:6803/1995005420] [v2:172.21.15.32:6804/1995005420,v1:172.21.15.32:6805/1995005420] exists,up 3a71e039-95f2-441c-86d2-65a3c98560e4 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/2541914444,v1:172.21.15.40:6801/2541914444] [v2:172.21.15.40:6802/2541914444,v1:172.21.15.40:6803/2541914444] exists,up 780165d2-56a4-4fee-98a0-bc98ba8c9ef7 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/3950848968,v1:172.21.15.64:6801/3950848968] [v2:172.21.15.64:6802/3950848968,v1:172.21.15.64:6803/3950848968] exists,up 50d71493-93bf-4dfe-ac4b-6a378dd2e244 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6810/2339361307,v1:172.21.15.32:6811/2339361307] [v2:172.21.15.32:6812/2339361307,v1:172.21.15.32:6813/2339361307] exists,up 253ddac0-f39a-459c-bef1-2d3bd28a2b2f 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/3289252322,v1:172.21.15.40:6809/3289252322] [v2:172.21.15.40:6810/3289252322,v1:172.21.15.40:6811/3289252322] exists,up c0f898fa-45b8-4ebd-bf04-b5758d43461e 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/1980421875,v1:172.21.15.64:6809/1980421875] [v2:172.21.15.64:6810/1980421875,v1:172.21.15.64:6811/1980421875] exists,up 9de82db1-b3dc-4181-9d27-fa440325720c 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6818/3191318575,v1:172.21.15.32:6819/3191318575] [v2:172.21.15.32:6820/3191318575,v1:172.21.15.32:6821/3191318575] exists,up ea694844-38fa-4af5-842f-1d69ba8d0679 2024-04-23T18:14:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6816/142928766,v1:172.21.15.40:6817/142928766] [v2:172.21.15.40:6818/142928766,v1:172.21.15.40:6819/142928766] exists,up c6eaac83-e79b-4a1e-8ab6-31b047aca03b 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/3639114663,v1:172.21.15.64:6817/3639114663] [v2:172.21.15.64:6818/3639114663,v1:172.21.15.64:6819/3639114663] exists,up bb2a922c-0367-4bc6-8556-983cf6b11b20 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6826/3112793650,v1:172.21.15.32:6827/3112793650] [v2:172.21.15.32:6828/3112793650,v1:172.21.15.32:6829/3112793650] exists,up 2db0b880-e61e-46ca-9826-421246e440fa 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6824/4006751902,v1:172.21.15.40:6825/4006751902] [v2:172.21.15.40:6826/4006751902,v1:172.21.15.40:6827/4006751902] exists,up b2116ca5-2fe8-405c-bfeb-0eac6b746fbc 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/4083016219,v1:172.21.15.64:6825/4083016219] [v2:172.21.15.64:6826/4083016219,v1:172.21.15.64:6827/4083016219] exists,up 4d5a0032-0c95-4c63-b06a-340f80f978de 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.a [2,11] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.c [2,1] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.1d [8,11] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.1f [8,4] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.3 [4,9] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.e [5,9] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.2c [4,9] 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1952921207 expires 2024-04-24T18:05:55.984321+0000 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/3732407927 expires 2024-04-24T18:05:55.984321+0000 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/2689415742 expires 2024-04-24T18:05:55.984321+0000 2024-04-23T18:14:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/2434896390 expires 2024-04-24T18:05:55.984321+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/776389606 expires 2024-04-24T18:05:28.817677+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/2689415742 expires 2024-04-24T18:05:55.984321+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1265262618 expires 2024-04-24T18:05:28.817677+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/3094904085 expires 2024-04-24T18:05:28.817677+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/3656601632 expires 2024-04-24T18:05:28.817677+0000 2024-04-23T18:14:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/3656601632 expires 2024-04-24T18:05:28.817677+0000 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.393+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 msgr2=0x7f0bd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.393+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 0x7f0bd40a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0be8103b20 tx=0x7f0be4009260 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.393+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 msgr2=0x7f0be8199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.393+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b580 tx=0x7f0bd800fa00 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 shutdown_connections 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be8104660 0x7f0be81a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0be8102ac0 0x7f0be8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 --2- 172.21.15.32:0/610142906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0be8103cc0 0x7f0be8199110 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 >> 172.21.15.32:0/610142906 conn(0x7f0be80fe060 msgr2=0x7f0be8107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 shutdown_connections 2024-04-23T18:14:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:54.394+0000 7f0bf0269700 1 -- 172.21.15.32:0/610142906 wait complete. 2024-04-23T18:14:54.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:53 smithi064 ceph-mon[33062]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:55.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:54 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/610142906' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:14:55.208 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-23T18:14:55.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.334+0000 7f62b0f68640 1 -- 172.21.15.32:0/905149041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac103b60 msgr2=0x7f62ac10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.334+0000 7f62b0f68640 1 --2- 172.21.15.32:0/905149041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac103b60 0x7f62ac10a3f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009ca0 tx=0x7f62a402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 -- 172.21.15.32:0/905149041 shutdown_connections 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- 172.21.15.32:0/905149041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac103b60 0x7f62ac10a3f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- 172.21.15.32:0/905149041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac1031a0 0x7f62ac103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- 172.21.15.32:0/905149041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac101fb0 0x7f62ac1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 -- 172.21.15.32:0/905149041 >> 172.21.15.32:0/905149041 conn(0x7f62ac0fd540 msgr2=0x7f62ac0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 -- 172.21.15.32:0/905149041 shutdown_connections 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 -- 172.21.15.32:0/905149041 wait complete. 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 Processor -- start 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 -- start start 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac101fb0 0x7f62ac198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.335+0000 7f62b0f68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac103b60 0x7f62ac19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62b0f68640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10db40 con 0x7f62ac1031a0 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62b0f68640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10d9c0 con 0x7f62ac103b60 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62b0f68640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10dcc0 con 0x7f62ac101fb0 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58402/0 (socket says 172.21.15.32:58402) 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 -- 172.21.15.32:0/1057666569 learned_addr learned my addr 172.21.15.32:0/1057666569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62aa59c640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac101fb0 0x7f62ac198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62aad9d640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac103b60 0x7f62ac19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 -- 172.21.15.32:0/1057666569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac101fb0 msgr2=0x7f62ac198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac101fb0 0x7f62ac198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 -- 172.21.15.32:0/1057666569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac103b60 msgr2=0x7f62ac19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac103b60 0x7f62ac19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 -- 172.21.15.32:0/1057666569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac19fd80 con 0x7f62ac1031a0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a3fff640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f629400d770 tx=0x7f629400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294017070 con 0x7f62ac1031a0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62940054d0 con 0x7f62ac1031a0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac1a0070 con 0x7f62ac1031a0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294004c30 con 0x7f62ac1031a0 2024-04-23T18:14:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.336+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac1a0570 con 0x7f62ac1031a0 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.337+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ac109270 con 0x7f62ac1031a0 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.339+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6294004e10 con 0x7f62ac1031a0 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.339+0000 7f62a37fe640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 0x7f62980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.339+0000 7f62aa59c640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 0x7f62980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.339+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f62940cfe90 con 0x7f62ac1031a0 2024-04-23T18:14:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.340+0000 7f62aa59c640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 0x7f62980a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f62ac103000 tx=0x7f629c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.344+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f629409c8a0 con 0x7f62ac1031a0 2024-04-23T18:14:55.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:54 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/610142906' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:14:55.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:54 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/610142906' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:14:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.538+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62ac062b50 con 0x7f62ac1031a0 2024-04-23T18:14:55.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.540+0000 7f62a37fe640 1 -- 172.21.15.32:0/1057666569 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7f629409c1a0 con 0x7f62ac1031a0 2024-04-23T18:14:55.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:55.534 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 msgr2=0x7f62980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 0x7f62980a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f62ac103000 tx=0x7f629c00b390 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 msgr2=0x7f62ac1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f629400d770 tx=0x7f629400dc30 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 shutdown_connections 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f62980a3bb0 0x7f62980a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ac103b60 0x7f62ac19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62ac1031a0 0x7f62ac1986a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 --2- 172.21.15.32:0/1057666569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62ac101fb0 0x7f62ac198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.543+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 >> 172.21.15.32:0/1057666569 conn(0x7f62ac0fd540 msgr2=0x7f62ac106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.544+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 shutdown_connections 2024-04-23T18:14:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.544+0000 7f62b0f68640 1 -- 172.21.15.32:0/1057666569 wait complete. 2024-04-23T18:14:55.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:55.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 -- 172.21.15.32:0/505005731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c1029a0 msgr2=0x7fde4c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 --2- 172.21.15.32:0/505005731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c1029a0 0x7fde4c1092c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fde44009ca0 tx=0x7fde4402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 -- 172.21.15.32:0/505005731 shutdown_connections 2024-04-23T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 --2- 172.21.15.32:0/505005731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c1029a0 0x7fde4c1092c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 --2- 172.21.15.32:0/505005731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde4c101fe0 0x7fde4c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 --2- 172.21.15.32:0/505005731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c100e90 0x7fde4c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 -- 172.21.15.32:0/505005731 >> 172.21.15.32:0/505005731 conn(0x7fde4c0fc420 msgr2=0x7fde4c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 -- 172.21.15.32:0/505005731 shutdown_connections 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.665+0000 7fde517e8640 1 -- 172.21.15.32:0/505005731 wait complete. 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 Processor -- start 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 -- start start 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c101fe0 0x7fde4c19f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde4c1029a0 0x7fde4c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10c9f0 con 0x7fde4c100e90 2024-04-23T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10c870 con 0x7fde4c1029a0 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde517e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10cb70 con 0x7fde4c101fe0 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde4affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde4affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58420/0 (socket says 172.21.15.32:58420) 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.666+0000 7fde4affd640 1 -- 172.21.15.32:0/3278219821 learned_addr learned my addr 172.21.15.32:0/3278219821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4a7fc640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c101fe0 0x7fde4c19f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 -- 172.21.15.32:0/3278219821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c101fe0 msgr2=0x7fde4c19f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c101fe0 0x7fde4c19f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 -- 172.21.15.32:0/3278219821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde4c1029a0 msgr2=0x7fde4c1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde4c1029a0 0x7fde4c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 -- 172.21.15.32:0/3278219821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4c1a1f90 con 0x7fde4c100e90 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde4affd640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00d770 tx=0x7fde3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c017070 con 0x7fde4c100e90 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4c1a2220 con 0x7fde4c100e90 2024-04-23T18:14:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3c0054d0 con 0x7fde4c100e90 2024-04-23T18:14:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c004c30 con 0x7fde4c100e90 2024-04-23T18:14:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.667+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde4c1a2720 con 0x7fde4c100e90 2024-04-23T18:14:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.668+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4c1081a0 con 0x7fde4c100e90 2024-04-23T18:14:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.669+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fde3c004dd0 con 0x7fde4c100e90 2024-04-23T18:14:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.670+0000 7fde33fff640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 0x7fde340a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.670+0000 7fde4a7fc640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 0x7fde340a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.670+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fde3c0cff90 con 0x7fde4c100e90 2024-04-23T18:14:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.670+0000 7fde4a7fc640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 0x7fde340a60a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fde400058c0 tx=0x7fde40005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.674+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde3c09c940 con 0x7fde4c100e90 2024-04-23T18:14:55.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.859+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde4c062c50 con 0x7fde4c100e90 2024-04-23T18:14:55.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.859+0000 7fde33fff640 1 -- 172.21.15.32:0/3278219821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fde3c09c240 con 0x7fde4c100e90 2024-04-23T18:14:55.854 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:14:55.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 msgr2=0x7fde340a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 0x7fde340a60a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fde400058c0 tx=0x7fde40005850 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 msgr2=0x7fde4c069dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00d770 tx=0x7fde3c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 shutdown_connections 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fde340a3be0 0x7fde340a60a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde4c1029a0 0x7fde4c1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde4c101fe0 0x7fde4c19f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 --2- 172.21.15.32:0/3278219821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde4c100e90 0x7fde4c069dc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.861+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 >> 172.21.15.32:0/3278219821 conn(0x7fde4c0fc420 msgr2=0x7fde4c105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.862+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 shutdown_connections 2024-04-23T18:14:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.862+0000 7fde517e8640 1 -- 172.21.15.32:0/3278219821 wait complete. 2024-04-23T18:14:55.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:14:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.978+0000 7fbc207ad640 1 -- 172.21.15.32:0/349705872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18102980 msgr2=0x7fbc181092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.978+0000 7fbc207ad640 1 --2- 172.21.15.32:0/349705872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18102980 0x7fbc181092a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009ca0 tx=0x7fbc1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- 172.21.15.32:0/349705872 shutdown_connections 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- 172.21.15.32:0/349705872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18102980 0x7fbc181092a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- 172.21.15.32:0/349705872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- 172.21.15.32:0/349705872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18100e70 0x7fbc18101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- 172.21.15.32:0/349705872 >> 172.21.15.32:0/349705872 conn(0x7fbc180fc3c0 msgr2=0x7fbc180fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- 172.21.15.32:0/349705872 shutdown_connections 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- 172.21.15.32:0/349705872 wait complete. 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 Processor -- start 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- start start 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18102980 0x7fbc1819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810ca40 con 0x7fbc18100e70 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810c8c0 con 0x7fbc18102980 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc207ad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810cbc0 con 0x7fbc18101fc0 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc1e549640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc1e549640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58440/0 (socket says 172.21.15.32:58440) 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.979+0000 7fbc1e549640 1 -- 172.21.15.32:0/3237504559 learned_addr learned my addr 172.21.15.32:0/3237504559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1dd48640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1ed4a640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18102980 0x7fbc1819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 -- 172.21.15.32:0/3237504559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 msgr2=0x7fbc18198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 -- 172.21.15.32:0/3237504559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18102980 msgr2=0x7fbc1819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18102980 0x7fbc1819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 -- 172.21.15.32:0/3237504559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1819fe40 con 0x7fbc18100e70 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1dd48640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc1e549640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400d950 tx=0x7fbc1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14017070 con 0x7fbc18100e70 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc14009c30 con 0x7fbc18100e70 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc181a0130 con 0x7fbc18100e70 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1400f620 con 0x7fbc18100e70 2024-04-23T18:14:55.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.980+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc181a0700 con 0x7fbc18100e70 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.981+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc181a0960 con 0x7fbc18100e70 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.983+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc14010460 con 0x7fbc18100e70 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.984+0000 7fbc0f7fe640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 0x7fbc040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.984+0000 7fbc1dd48640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 0x7fbc040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.984+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbc140d00f0 con 0x7fbc18100e70 2024-04-23T18:14:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.984+0000 7fbc1dd48640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 0x7fbc040a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006fd0 tx=0x7fbc08008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:55.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:55.987+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc1409caa0 con 0x7fbc18100e70 2024-04-23T18:14:56.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.175+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc181a0b30 con 0x7fbc18100e70 2024-04-23T18:14:56.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.176+0000 7fbc0f7fe640 1 -- 172.21.15.32:0/3237504559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fbc14009de0 con 0x7fbc18100e70 2024-04-23T18:14:56.170 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 msgr2=0x7fbc040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 0x7fbc040a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006fd0 tx=0x7fbc08008040 comp rx=0 tx=0).stop 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 msgr2=0x7fbc18198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400d950 tx=0x7fbc1400de10 comp rx=0 tx=0).stop 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 shutdown_connections 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fbc040a3bb0 0x7fbc040a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc18102980 0x7fbc1819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc18101fc0 0x7fbc18198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 --2- 172.21.15.32:0/3237504559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc18100e70 0x7fbc18198220 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 >> 172.21.15.32:0/3237504559 conn(0x7fbc180fc3c0 msgr2=0x7fbc180fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 shutdown_connections 2024-04-23T18:14:56.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.178+0000 7fbc207ad640 1 -- 172.21.15.32:0/3237504559 wait complete. 2024-04-23T18:14:56.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:55 smithi032 ceph-mon[29221]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:56.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:55 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1057666569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:56.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:55 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3278219821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:56.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:14:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 -- 172.21.15.32:0/4162311282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0100e90 msgr2=0x7f8fe01012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/4162311282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0100e90 0x7f8fe01012b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009900 tx=0x7f8fc802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 -- 172.21.15.32:0/4162311282 shutdown_connections 2024-04-23T18:14:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/4162311282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 0x7f8fe01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/4162311282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/4162311282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0100e90 0x7f8fe01012b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 -- 172.21.15.32:0/4162311282 >> 172.21.15.32:0/4162311282 conn(0x7f8fe00fc420 msgr2=0x7f8fe00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 -- 172.21.15.32:0/4162311282 shutdown_connections 2024-04-23T18:14:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.294+0000 7f8fe49a2640 1 -- 172.21.15.32:0/4162311282 wait complete. 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 Processor -- start 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 -- start start 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0100e90 0x7f8fe01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 0x7f8fe019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe010db00 con 0x7f8fe0101fe0 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe010d980 con 0x7f8fe0100e90 2024-04-23T18:14:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fe49a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe010dc80 con 0x7f8fe01029a0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58462/0 (socket says 172.21.15.32:58462) 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 -- 172.21.15.32:0/967567667 learned_addr learned my addr 172.21.15.32:0/967567667 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fded9d640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 0x7f8fe019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 -- 172.21.15.32:0/967567667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 msgr2=0x7f8fe019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fde59c640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0100e90 0x7f8fe01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 0x7f8fe019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 -- 172.21.15.32:0/967567667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0100e90 msgr2=0x7f8fe01981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0100e90 0x7f8fe01981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.295+0000 7f8fddd9b640 1 -- 172.21.15.32:0/967567667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe019fe00 con 0x7f8fe0101fe0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fddd9b640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000d950 tx=0x7f8fd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0017020 con 0x7f8fe0101fe0 2024-04-23T18:14:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd0009c30 con 0x7f8fe0101fe0 2024-04-23T18:14:56.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe01a00f0 con 0x7f8fe0101fe0 2024-04-23T18:14:56.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000f6b0 con 0x7f8fe0101fe0 2024-04-23T18:14:56.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.296+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01a0640 con 0x7f8fe0101fe0 2024-04-23T18:14:56.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.297+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe0108210 con 0x7f8fe0101fe0 2024-04-23T18:14:56.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.298+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8fd001a020 con 0x7f8fe0101fe0 2024-04-23T18:14:56.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.298+0000 7f8fd77fe640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 0x7f8fcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.299+0000 7f8fde59c640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 0x7f8fcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.299+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8fd00d0150 con 0x7f8fe0101fe0 2024-04-23T18:14:56.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.299+0000 7f8fde59c640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 0x7f8fcc0a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8005c40 tx=0x7f8fc8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.303+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fd0009de0 con 0x7f8fe0101fe0 2024-04-23T18:14:56.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:55 smithi040 ceph-mon[32626]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:56.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:55 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1057666569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:56.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:55 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3278219821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:56.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:55 smithi064 ceph-mon[33062]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:14:56.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:55 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1057666569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:56.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:55 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3278219821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:56.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.490+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8fe0062b50 con 0x7f8fe0101fe0 2024-04-23T18:14:56.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.491+0000 7f8fd77fe640 1 -- 172.21.15.32:0/967567667 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f8fd009cb60 con 0x7f8fe0101fe0 2024-04-23T18:14:56.485 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 msgr2=0x7f8fcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 0x7f8fcc0a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8005c40 tx=0x7f8fc8000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 msgr2=0x7f8fe0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000d950 tx=0x7f8fd000de10 comp rx=0 tx=0).stop 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 shutdown_connections 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fe01029a0 0x7f8fe019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe0101fe0 0x7f8fe0198720 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8fcc0a3bb0 0x7f8fcc0a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 --2- 172.21.15.32:0/967567667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe0100e90 0x7f8fe01981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 >> 172.21.15.32:0/967567667 conn(0x7f8fe00fc420 msgr2=0x7f8fe01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 shutdown_connections 2024-04-23T18:14:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.493+0000 7f8fe49a2640 1 -- 172.21.15.32:0/967567667 wait complete. 2024-04-23T18:14:56.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 -- 172.21.15.32:0/681952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f01032b0 msgr2=0x7f80f0103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 --2- 172.21.15.32:0/681952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f01032b0 0x7f80f0103730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009930 tx=0x7f80d802a420 comp rx=0 tx=0).stop 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 -- 172.21.15.32:0/681952152 shutdown_connections 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 --2- 172.21.15.32:0/681952152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f0103c70 0x7f80f010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 --2- 172.21.15.32:0/681952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f01032b0 0x7f80f0103730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 --2- 172.21.15.32:0/681952152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01002e0 0x7f80f0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 -- 172.21.15.32:0/681952152 >> 172.21.15.32:0/681952152 conn(0x7f80f00fbc10 msgr2=0x7f80f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 -- 172.21.15.32:0/681952152 shutdown_connections 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.606+0000 7f80f48bb640 1 -- 172.21.15.32:0/681952152 wait complete. 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 Processor -- start 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 -- start start 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f01002e0 0x7f80f0102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01032b0 0x7f80f0101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010db10 con 0x7f80f0103c70 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010d990 con 0x7f80f01032b0 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80f48bb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010dc90 con 0x7f80f01002e0 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58476/0 (socket says 172.21.15.32:58476) 2024-04-23T18:14:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 -- 172.21.15.32:0/4246655468 learned_addr learned my addr 172.21.15.32:0/4246655468 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80e7fff640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01032b0 0x7f80f0101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80ee59c640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f01002e0 0x7f80f0102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 -- 172.21.15.32:0/4246655468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f01002e0 msgr2=0x7f80f0102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f01002e0 0x7f80f0102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 -- 172.21.15.32:0/4246655468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01032b0 msgr2=0x7f80f0101260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01032b0 0x7f80f0101260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 -- 172.21.15.32:0/4246655468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f0102030 con 0x7f80f0103c70 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.607+0000 7f80eed9d640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f80e800cce0 tx=0x7f80e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.608+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e800f450 con 0x7f80f0103c70 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.608+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f01a2290 con 0x7f80f0103c70 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.608+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e800faa0 con 0x7f80f0103c70 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.608+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8018b70 con 0x7f80f0103c70 2024-04-23T18:14:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.608+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f01a2760 con 0x7f80f0103c70 2024-04-23T18:14:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.609+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f0069250 con 0x7f80f0103c70 2024-04-23T18:14:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.610+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f80e8018d10 con 0x7f80f0103c70 2024-04-23T18:14:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.610+0000 7f80e77fe640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 0x7f80dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.611+0000 7f80ee59c640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 0x7f80dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.611+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f80e80d0870 con 0x7f80f0103c70 2024-04-23T18:14:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.611+0000 7f80ee59c640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 0x7f80dc0a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f80e0005fd0 tx=0x7f80e0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.614+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80e8014030 con 0x7f80f0103c70 2024-04-23T18:14:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.839+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80f0109460 con 0x7f80f0103c70 2024-04-23T18:14:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.839+0000 7f80e77fe640 1 -- 172.21.15.32:0/4246655468 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7f80e809d310 con 0x7f80f0103c70 2024-04-23T18:14:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:56.834 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:14:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 msgr2=0x7f80dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 0x7f80dc0a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f80e0005fd0 tx=0x7f80e0005c70 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 msgr2=0x7f80f01017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f80e800cce0 tx=0x7f80e800a620 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 shutdown_connections 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.841+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f0103c70 0x7f80f01017a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f80dc0a3bb0 0x7f80dc0a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80f01032b0 0x7f80f0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 --2- 172.21.15.32:0/4246655468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80f01002e0 0x7f80f0102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 >> 172.21.15.32:0/4246655468 conn(0x7f80f00fbc10 msgr2=0x7f80f0106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 shutdown_connections 2024-04-23T18:14:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.842+0000 7f80f48bb640 1 -- 172.21.15.32:0/4246655468 wait complete. 2024-04-23T18:14:56.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- 172.21.15.32:0/505002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41031a0 msgr2=0x7f46a4103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 --2- 172.21.15.32:0/505002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41031a0 0x7f46a4103620 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4698009900 tx=0x7f469802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- 172.21.15.32:0/505002029 shutdown_connections 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 --2- 172.21.15.32:0/505002029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 0x7f46a410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 --2- 172.21.15.32:0/505002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41031a0 0x7f46a4103620 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 --2- 172.21.15.32:0/505002029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a4101fb0 0x7f46a41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- 172.21.15.32:0/505002029 >> 172.21.15.32:0/505002029 conn(0x7f46a40fd540 msgr2=0x7f46a40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- 172.21.15.32:0/505002029 shutdown_connections 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- 172.21.15.32:0/505002029 wait complete. 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 Processor -- start 2024-04-23T18:14:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.955+0000 7f46a9bad640 1 -- start start 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a41031a0 0x7f46a4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 0x7f46a4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410db50 con 0x7f46a4101fb0 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410d9d0 con 0x7f46a4103b60 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410dcd0 con 0x7f46a41031a0 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59784/0 (socket says 172.21.15.32:59784) 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 -- 172.21.15.32:0/2835708939 learned_addr learned my addr 172.21.15.32:0/2835708939 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a3fff640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 0x7f46a4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 -- 172.21.15.32:0/2835708939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a41031a0 msgr2=0x7f46a4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a41031a0 0x7f46a4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 -- 172.21.15.32:0/2835708939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 msgr2=0x7f46a4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 0x7f46a4074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 -- 172.21.15.32:0/2835708939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a4075110 con 0x7f46a4101fb0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a37fe640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f469400b450 tx=0x7f469400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694015020 con 0x7f46a4101fb0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4694011d40 con 0x7f46a4101fb0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469400f340 con 0x7f46a4101fb0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.956+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a41a22e0 con 0x7f46a4101fb0 2024-04-23T18:14:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.957+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a41a28b0 con 0x7f46a4101fb0 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.957+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a41092d0 con 0x7f46a4101fb0 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.959+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4694018020 con 0x7f46a4101fb0 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.959+0000 7f46a0ff9640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.959+0000 7f46a2ffd640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.960+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f469409a8a0 con 0x7f46a4101fb0 2024-04-23T18:14:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.960+0000 7f46a2ffd640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f46980096c0 tx=0x7f4698000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:56.965+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f469401f020 con 0x7f46a4101fb0 2024-04-23T18:14:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.156+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46a402d830 con 0x7f46a4101fb0 2024-04-23T18:14:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.158+0000 7f46a0ff9640 1 -- 172.21.15.32:0/2835708939 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7f469400d040 con 0x7f46a4101fb0 2024-04-23T18:14:57.152 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:57.152 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 msgr2=0x7f468c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f46980096c0 tx=0x7f4698000bc0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 msgr2=0x7f46a4075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f469400b450 tx=0x7f469400b910 comp rx=0 tx=0).stop 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 shutdown_connections 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4103b60 0x7f46a4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46a41031a0 0x7f46a4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 --2- 172.21.15.32:0/2835708939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4101fb0 0x7f46a4075d00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 >> 172.21.15.32:0/2835708939 conn(0x7f46a40fd540 msgr2=0x7f46a41064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 shutdown_connections 2024-04-23T18:14:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.160+0000 7f46a9bad640 1 -- 172.21.15.32:0/2835708939 wait complete. 2024-04-23T18:14:57.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:56 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3237504559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:56 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/967567667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:56 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4246655468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:57.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:57.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:14:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 -- 172.21.15.32:0/793203627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88103b60 msgr2=0x7f4f8810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 --2- 172.21.15.32:0/793203627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88103b60 0x7f4f8810a3f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009ca0 tx=0x7f4f8002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 -- 172.21.15.32:0/793203627 shutdown_connections 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 --2- 172.21.15.32:0/793203627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88103b60 0x7f4f8810a3f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 --2- 172.21.15.32:0/793203627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f881031a0 0x7f4f88103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 --2- 172.21.15.32:0/793203627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f88101fb0 0x7f4f881023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 -- 172.21.15.32:0/793203627 >> 172.21.15.32:0/793203627 conn(0x7f4f880fd520 msgr2=0x7f4f880ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 -- 172.21.15.32:0/793203627 shutdown_connections 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.276+0000 7f4f8c820640 1 -- 172.21.15.32:0/793203627 wait complete. 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 Processor -- start 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 -- start start 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f881031a0 0x7f4f881986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88103b60 0x7f4f8819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8810daf0 con 0x7f4f88101fb0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8810d970 con 0x7f4f88103b60 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8c820640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8810dc70 con 0x7f4f881031a0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59806/0 (socket says 172.21.15.32:59806) 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.277+0000 7f4f8659c640 1 -- 172.21.15.32:0/2840551292 learned_addr learned my addr 172.21.15.32:0/2840551292 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f7ffff640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f881031a0 0x7f4f881986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f86d9d640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88103b60 0x7f4f8819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 -- 172.21.15.32:0/2840551292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f881031a0 msgr2=0x7f4f881986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f881031a0 0x7f4f881986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 -- 172.21.15.32:0/2840551292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88103b60 msgr2=0x7f4f8819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88103b60 0x7f4f8819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 -- 172.21.15.32:0/2840551292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8819fdb0 con 0x7f4f88101fb0 2024-04-23T18:14:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8659c640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c970 tx=0x7f4f7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004220 con 0x7f4f88101fb0 2024-04-23T18:14:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780043c0 con 0x7f4f88101fb0 2024-04-23T18:14:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f881a00a0 con 0x7f4f88101fb0 2024-04-23T18:14:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004a00 con 0x7f4f88101fb0 2024-04-23T18:14:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.278+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f881a05f0 con 0x7f4f88101fb0 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.279+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f881093a0 con 0x7f4f88101fb0 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.280+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f780079b0 con 0x7f4f88101fb0 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.281+0000 7f4f7f7fe640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 0x7f4f740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.281+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4f780d00e0 con 0x7f4f88101fb0 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.281+0000 7f4f7ffff640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 0x7f4f740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.281+0000 7f4f7ffff640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 0x7f4f740a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4f881996a0 tx=0x7f4f7000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.285+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f78014030 con 0x7f4f88101fb0 2024-04-23T18:14:57.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:56 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3237504559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:56 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/967567667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:56 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4246655468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:57.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:56 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3237504559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:56 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/967567667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:57.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:56 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4246655468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.470+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f88062b50 con 0x7f4f88101fb0 2024-04-23T18:14:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.471+0000 7f4f7f7fe640 1 -- 172.21.15.32:0/2840551292 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f4f7809caf0 con 0x7f4f88101fb0 2024-04-23T18:14:57.465 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:14:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 msgr2=0x7f4f740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 0x7f4f740a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4f881996a0 tx=0x7f4f7000b410 comp rx=0 tx=0).stop 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 msgr2=0x7f4f88198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c970 tx=0x7f4f7800ce30 comp rx=0 tx=0).stop 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 shutdown_connections 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f4f740a3c30 0x7f4f740a60f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88103b60 0x7f4f8819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f881031a0 0x7f4f881986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 --2- 172.21.15.32:0/2840551292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f88101fb0 0x7f4f88198190 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 >> 172.21.15.32:0/2840551292 conn(0x7f4f880fd520 msgr2=0x7f4f88106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.472+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 shutdown_connections 2024-04-23T18:14:57.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.473+0000 7f4f8c820640 1 -- 172.21.15.32:0/2840551292 wait complete. 2024-04-23T18:14:57.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:14:57.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- 172.21.15.32:0/1542640503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102040 msgr2=0x7f56281024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 --2- 172.21.15.32:0/1542640503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102040 0x7f56281024c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5618009900 tx=0x7f561802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:57.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- 172.21.15.32:0/1542640503 shutdown_connections 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 --2- 172.21.15.32:0/1542640503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102a00 0x7f5628109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 --2- 172.21.15.32:0/1542640503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102040 0x7f56281024c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 --2- 172.21.15.32:0/1542640503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 0x7f5628101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- 172.21.15.32:0/1542640503 >> 172.21.15.32:0/1542640503 conn(0x7f56280fc400 msgr2=0x7f56280fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- 172.21.15.32:0/1542640503 shutdown_connections 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- 172.21.15.32:0/1542640503 wait complete. 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 Processor -- start 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.593+0000 7f562fd76640 1 -- start start 2024-04-23T18:14:57.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 0x7f56281981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102040 0x7f5628198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f562810db70 con 0x7f5628102a00 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f562810d9f0 con 0x7f5628100e60 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f562810dcf0 con 0x7f5628102040 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59816/0 (socket says 172.21.15.32:59816) 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 -- 172.21.15.32:0/3842304543 learned_addr learned my addr 172.21.15.32:0/3842304543 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562d311640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102040 0x7f5628198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562db12640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 0x7f56281981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 -- 172.21.15.32:0/3842304543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102040 msgr2=0x7f5628198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102040 0x7f5628198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 -- 172.21.15.32:0/3842304543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 msgr2=0x7f56281981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 0x7f56281981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 -- 172.21.15.32:0/3842304543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562819fe00 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562e313640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f562000b970 tx=0x7f562000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620019070 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56281a00f0 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5620014440 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200075d0 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.594+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56281a06c0 con 0x7f5628102a00 2024-04-23T18:14:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.595+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5628108250 con 0x7f5628102a00 2024-04-23T18:14:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.596+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f562001c070 con 0x7f5628102a00 2024-04-23T18:14:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.597+0000 7f561effd640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 0x7f56140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.597+0000 7f562db12640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 0x7f56140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.597+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f56200cfa10 con 0x7f5628102a00 2024-04-23T18:14:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.598+0000 7f562db12640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 0x7f56140a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.602+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f562009c420 con 0x7f5628102a00 2024-04-23T18:14:57.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.785+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5628062b50 con 0x7f5628102a00 2024-04-23T18:14:57.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.786+0000 7f561effd640 1 -- 172.21.15.32:0/3842304543 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f562009bd20 con 0x7f5628102a00 2024-04-23T18:14:57.780 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 msgr2=0x7f56140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 0x7f56140a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624005850 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 msgr2=0x7f562819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f562000b970 tx=0x7f562000be30 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 shutdown_connections 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5628102a00 0x7f562819f790 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5628102040 0x7f5628198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f56140a3bb0 0x7f56140a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.787+0000 7f562fd76640 1 --2- 172.21.15.32:0/3842304543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5628100e60 0x7f56281981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.788+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 >> 172.21.15.32:0/3842304543 conn(0x7f56280fc400 msgr2=0x7f5628105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.788+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 shutdown_connections 2024-04-23T18:14:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.788+0000 7f562fd76640 1 -- 172.21.15.32:0/3842304543 wait complete. 2024-04-23T18:14:57.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:14:57.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 -- 172.21.15.32:0/3146811002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028107c70 msgr2=0x7f002810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 --2- 172.21.15.32:0/3146811002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028107c70 0x7f002810a0c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0018009ca0 tx=0x7f001802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 -- 172.21.15.32:0/3146811002 shutdown_connections 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 --2- 172.21.15.32:0/3146811002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028107c70 0x7f002810a0c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 --2- 172.21.15.32:0/3146811002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028105340 0x7f0028107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 --2- 172.21.15.32:0/3146811002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028102a10 0x7f0028104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 -- 172.21.15.32:0/3146811002 >> 172.21.15.32:0/3146811002 conn(0x7f00280fc420 msgr2=0x7f00280fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 -- 172.21.15.32:0/3146811002 shutdown_connections 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.903+0000 7f002d4cb640 1 -- 172.21.15.32:0/3146811002 wait complete. 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 Processor -- start 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 -- start start 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 0x7f0028198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028107c70 0x7f002819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f002810dad0 con 0x7f0028102a10 2024-04-23T18:14:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f002810d950 con 0x7f0028105340 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f002d4cb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f002810dc50 con 0x7f0028107c70 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f0026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f0026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59840/0 (socket says 172.21.15.32:59840) 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f0026ffd640 1 -- 172.21.15.32:0/1163667072 learned_addr learned my addr 172.21.15.32:0/1163667072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.904+0000 7f00277fe640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028107c70 0x7f002819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f00267fc640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 0x7f0028198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 -- 172.21.15.32:0/1163667072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028107c70 msgr2=0x7f002819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028107c70 0x7f002819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 -- 172.21.15.32:0/1163667072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 msgr2=0x7f0028198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 0x7f0028198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 -- 172.21.15.32:0/1163667072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002819fdf0 con 0x7f0028102a10 2024-04-23T18:14:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f00267fc640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 0x7f0028198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f0026ffd640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f002000e9f0 tx=0x7f002000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000cd80 con 0x7f0028102a10 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281a00e0 con 0x7f0028102a10 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00200108e0 con 0x7f0028102a10 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020018980 con 0x7f0028102a10 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.905+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281a0630 con 0x7f0028102a10 2024-04-23T18:14:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.906+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00280fdf10 con 0x7f0028102a10 2024-04-23T18:14:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.907+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0020010430 con 0x7f0028102a10 2024-04-23T18:14:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.908+0000 7f001ffff640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 0x7f00140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.908+0000 7f00267fc640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 0x7f00140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.908+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f00200d07b0 con 0x7f0028102a10 2024-04-23T18:14:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.908+0000 7f00267fc640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 0x7f00140a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f00281996e0 tx=0x7f0010005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:57.913+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f002009d160 con 0x7f0028102a10 2024-04-23T18:14:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.101+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0028062b50 con 0x7f0028102a10 2024-04-23T18:14:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.101+0000 7f001ffff640 1 -- 172.21.15.32:0/1163667072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f002009ca60 con 0x7f0028102a10 2024-04-23T18:14:58.096 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:14:58.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 msgr2=0x7f00140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 0x7f00140a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f00281996e0 tx=0x7f0010005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 msgr2=0x7f00281981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f002000e9f0 tx=0x7f002000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 shutdown_connections 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0028107c70 0x7f002819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f00140a3bb0 0x7f00140a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0028105340 0x7f0028198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 --2- 172.21.15.32:0/1163667072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0028102a10 0x7f00281981d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 >> 172.21.15.32:0/1163667072 conn(0x7f00280fc420 msgr2=0x7f0028103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 shutdown_connections 2024-04-23T18:14:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.103+0000 7f002d4cb640 1 -- 172.21.15.32:0/1163667072 wait complete. 2024-04-23T18:14:58.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:57 smithi032 ceph-mon[29221]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:58.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2835708939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2840551292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3842304543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- 172.21.15.32:0/1047053124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10107c90 msgr2=0x7f6e1010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 --2- 172.21.15.32:0/1047053124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10107c90 0x7f6e1010a0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009ca0 tx=0x7f6e0802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- 172.21.15.32:0/1047053124 shutdown_connections 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 --2- 172.21.15.32:0/1047053124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10107c90 0x7f6e1010a0e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 --2- 172.21.15.32:0/1047053124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10105360 0x7f6e10107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 --2- 172.21.15.32:0/1047053124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 0x7f6e10104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- 172.21.15.32:0/1047053124 >> 172.21.15.32:0/1047053124 conn(0x7f6e100fc460 msgr2=0x7f6e100fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- 172.21.15.32:0/1047053124 shutdown_connections 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- 172.21.15.32:0/1047053124 wait complete. 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 Processor -- start 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.217+0000 7f6e164cf640 1 -- start start 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 0x7f6e10102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10107c90 0x7f6e10100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1010db40 con 0x7f6e10105360 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1010d9c0 con 0x7f6e10107c90 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1010dcc0 con 0x7f6e10102a30 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59856/0 (socket says 172.21.15.32:59856) 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 -- 172.21.15.32:0/3444833151 learned_addr learned my addr 172.21.15.32:0/3444833151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0ffff640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 0x7f6e10102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e14a6c640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10107c90 0x7f6e10100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 -- 172.21.15.32:0/3444833151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 msgr2=0x7f6e10102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 0x7f6e10102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 -- 172.21.15.32:0/3444833151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10107c90 msgr2=0x7f6e10100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10107c90 0x7f6e10100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 -- 172.21.15.32:0/3444833151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e101017e0 con 0x7f6e10105360 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0f7fe640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000c970 tx=0x7f6e0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00007bb0 con 0x7f6e10105360 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e00007d50 con 0x7f6e10105360 2024-04-23T18:14:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.218+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e101a22d0 con 0x7f6e10105360 2024-04-23T18:14:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.219+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000f450 con 0x7f6e10105360 2024-04-23T18:14:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.219+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e101a2840 con 0x7f6e10105360 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.220+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e100fdf50 con 0x7f6e10105360 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.221+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6e00004f20 con 0x7f6e10105360 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.222+0000 7f6e0d7fa640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 0x7f6df80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.222+0000 7f6e0ffff640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 0x7f6df80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.222+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6e000d02a0 con 0x7f6e10105360 2024-04-23T18:14:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.222+0000 7f6e0ffff640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 0x7f6df80a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e0400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.226+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e00014030 con 0x7f6e10105360 2024-04-23T18:14:58.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:57 smithi040 ceph-mon[32626]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:58.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2835708939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2840551292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3842304543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:57 smithi064 ceph-mon[33062]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:14:58.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2835708939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2840551292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3842304543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:14:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.451+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e10062b50 con 0x7f6e10105360 2024-04-23T18:14:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.451+0000 7f6e0d7fa640 1 -- 172.21.15.32:0/3444833151 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7f6e0009c660 con 0x7f6e10105360 2024-04-23T18:14:58.446 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:58.446 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:13:10.043413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:14:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 msgr2=0x7f6df80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 0x7f6df80a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e0400b390 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 msgr2=0x7f6e101009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000c970 tx=0x7f6e0000ce30 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 shutdown_connections 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6df80a3c30 0x7f6df80a60f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.453+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e10107c90 0x7f6e10100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.454+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e10105360 0x7f6e101009e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.454+0000 7f6e164cf640 1 --2- 172.21.15.32:0/3444833151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e10102a30 0x7f6e10102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.454+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 >> 172.21.15.32:0/3444833151 conn(0x7f6e100fc460 msgr2=0x7f6e10103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.454+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 shutdown_connections 2024-04-23T18:14:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.454+0000 7f6e164cf640 1 -- 172.21.15.32:0/3444833151 wait complete. 2024-04-23T18:14:58.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- 172.21.15.32:0/1936658592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 msgr2=0x7fe90c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1936658592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c1077b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009900 tx=0x7fe8fc02a410 comp rx=0 tx=0).stop 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- 172.21.15.32:0/1936658592 shutdown_connections 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1936658592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 0x7fe90c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1936658592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c1077b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1936658592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 0x7fe90c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- 172.21.15.32:0/1936658592 >> 172.21.15.32:0/1936658592 conn(0x7fe90c0fc460 msgr2=0x7fe90c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- 172.21.15.32:0/1936658592 shutdown_connections 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- 172.21.15.32:0/1936658592 wait complete. 2024-04-23T18:14:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 Processor -- start 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 -- start start 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.568+0000 7fe9145a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 0x7fe90c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 0x7fe90c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10daf0 con 0x7fe90c1053c0 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10d970 con 0x7fe90c102a90 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10dc70 con 0x7fe90c107cf0 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59876/0 (socket says 172.21.15.32:59876) 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 -- 172.21.15.32:0/1790756944 learned_addr learned my addr 172.21.15.32:0/1790756944 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe912b3f640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 0x7fe90c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe91233e640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 0x7fe90c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 -- 172.21.15.32:0/1790756944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 msgr2=0x7fe90c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 0x7fe90c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 -- 172.21.15.32:0/1790756944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 msgr2=0x7fe90c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 0x7fe90c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 -- 172.21.15.32:0/1790756944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c19fe30 con 0x7fe90c1053c0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe911b3d640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006010 tx=0x7fe8fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc03f030 con 0x7fe90c1053c0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8fc031910 con 0x7fe90c1053c0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c1a00c0 con 0x7fe90c1053c0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc035070 con 0x7fe90c1053c0 2024-04-23T18:14:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.569+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c1a05b0 con 0x7fe90c1053c0 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.570+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c0fdf50 con 0x7fe90c1053c0 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.572+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe8fc042030 con 0x7fe90c1053c0 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.572+0000 7fe9037fe640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 0x7fe8f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.573+0000 7fe91233e640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 0x7fe8f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.573+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe8fc0f08c0 con 0x7fe90c1053c0 2024-04-23T18:14:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.573+0000 7fe91233e640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 0x7fe8f80a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe908009bd0 tx=0x7fe908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.577+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8fc0bcc80 con 0x7fe90c1053c0 2024-04-23T18:14:58.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.766+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe90c062b50 con 0x7fe90c1053c0 2024-04-23T18:14:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.767+0000 7fe9037fe640 1 -- 172.21.15.32:0/1790756944 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22533 (secure 0 0 0) 0x7fe8fc003e60 con 0x7fe90c1053c0 2024-04-23T18:14:58.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:14:58.762 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:13:57.185219+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 msgr2=0x7fe8f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 0x7fe8f80a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe908009bd0 tx=0x7fe908005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 msgr2=0x7fe90c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006010 tx=0x7fe8fc0042f0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 shutdown_connections 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe90c107cf0 0x7fe90c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c1053c0 0x7fe90c198750 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fe8f80a3bb0 0x7fe8f80a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 --2- 172.21.15.32:0/1790756944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe90c102a90 0x7fe90c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 >> 172.21.15.32:0/1790756944 conn(0x7fe90c0fc460 msgr2=0x7fe90c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 shutdown_connections 2024-04-23T18:14:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.770+0000 7fe9145a2640 1 -- 172.21.15.32:0/1790756944 wait complete. 2024-04-23T18:14:58.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:13:05.375853+0000', 'modified': '2024-04-23T18:13:10.043413+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24551, 'mds_1': 24515, 'mds_2': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/669117639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 669117639}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 669117639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/1068324383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1068324383}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1068324383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6837/1625538187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 1625538187}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 1625538187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.64:6839/1242971440', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 1242971440}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 1242971440}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6837/1154751886', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 1154751886}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 1154751886}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2479383965', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2479383965}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2479383965}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24551, 'qdb_cluster': [24499, 24515, 24551]} 2024-04-23T18:14:58.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:14:58.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:14:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 -- 172.21.15.32:0/2263832025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 msgr2=0x7f04c0103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 --2- 172.21.15.32:0/2263832025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0103750 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009900 tx=0x7f04b002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 -- 172.21.15.32:0/2263832025 shutdown_connections 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 --2- 172.21.15.32:0/2263832025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c0103c90 0x7f04c010a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 --2- 172.21.15.32:0/2263832025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0103750 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 --2- 172.21.15.32:0/2263832025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c01020e0 0x7f04c0102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.884+0000 7f04c7522640 1 -- 172.21.15.32:0/2263832025 >> 172.21.15.32:0/2263832025 conn(0x7f04c00fd650 msgr2=0x7f04c00ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- 172.21.15.32:0/2263832025 shutdown_connections 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- 172.21.15.32:0/2263832025 wait complete. 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 Processor -- start 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- start start 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 0x7f04c0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c0103c90 0x7f04c019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010dbb0 con 0x7f04c01032d0 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010da30 con 0x7f04c0103c90 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c7522640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010dd30 con 0x7f04c01020e0 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c4abd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c4abd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59888/0 (socket says 172.21.15.32:59888) 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c4abd640 1 -- 172.21.15.32:0/2435995666 learned_addr learned my addr 172.21.15.32:0/2435995666 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c0103c90 0x7f04c019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.885+0000 7f04c52be640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 0x7f04c0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 -- 172.21.15.32:0/2435995666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 msgr2=0x7f04c0198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 0x7f04c0198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 -- 172.21.15.32:0/2435995666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c0103c90 msgr2=0x7f04c019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c0103c90 0x7f04c019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 -- 172.21.15.32:0/2435995666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c019fe50 con 0x7f04c01032d0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c52be640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 0x7f04c0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c4abd640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04b0005940 tx=0x7f04b0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04b67fc640 1 -- 172.21.15.32:0/2435995666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b003f030 con 0x7f04c01032d0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04b67fc640 1 -- 172.21.15.32:0/2435995666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b0003aa0 con 0x7f04c01032d0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04b67fc640 1 -- 172.21.15.32:0/2435995666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0035070 con 0x7f04c01032d0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c01a00e0 con 0x7f04c01032d0 2024-04-23T18:14:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.886+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c01a05d0 con 0x7f04c01032d0 2024-04-23T18:14:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04b67fc640 1 -- 172.21.15.32:0/2435995666 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f04b0042030 con 0x7f04c01032d0 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 Processor -- start 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 -- start start 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0cd50 0x563cafb0d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb13fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cafa70c50 con 0x563cafb0bb80 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563cafa0d830 con 0x563cafb0d710 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04ca23e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563cafa6cc10 con 0x563cafb0cd50 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.889+0000 7f04c5abf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59904/0 (socket says 172.21.15.32:59904) 2024-04-23T18:14:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 -- 172.21.15.32:0/2839215221 learned_addr learned my addr 172.21.15.32:0/2839215221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c52be640 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0cd50 0x563cafb0d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04b67fc640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 0x7f04ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c4abd640 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb13fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 -- 172.21.15.32:0/2839215221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0cd50 msgr2=0x563cafb0d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0cd50 0x563cafb0d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 -- 172.21.15.32:0/2839215221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 msgr2=0x563cafb13fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb13fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 -- 172.21.15.32:0/2839215221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563caf7832d0 con 0x563cafb0bb80 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04b67fc640 1 -- 172.21.15.32:0/2435995666 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f04b00f0a70 con 0x7f04c01032d0 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 0x7f04ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f04b8010510 tx=0x7f04b8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8645af1078bc618b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04ab7fe640 1 -- 172.21.15.32:0/2839215221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8023070 con 0x563cafb0bb80 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04ab7fe640 1 -- 172.21.15.32:0/2839215221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b801dab0 con 0x563cafb0bb80 2024-04-23T18:14:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.890+0000 7f04ab7fe640 1 -- 172.21.15.32:0/2839215221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b801c820 con 0x563cafb0bb80 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- 172.21.15.32:0/2839215221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 msgr2=0x563cafb0bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f04b8010510 tx=0x7f04b8009f90 comp rx=0 tx=0).stop 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04c5abf640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 0x7f04ac0a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f04b8020000 tx=0x7f04b801b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- 172.21.15.32:0/2839215221 shutdown_connections 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb13fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0cd50 0x563cafb0d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 --2- 172.21.15.32:0/2839215221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb0bfa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- 172.21.15.32:0/2839215221 >> 172.21.15.32:0/2839215221 conn(0x563cafb07070 msgr2=0x563cafb07470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- 172.21.15.32:0/2839215221 shutdown_connections 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- 172.21.15.32:0/2839215221 wait complete. 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 -- start start 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.891+0000 7f04ca23e740 1 Processor -- start 2024-04-23T18:14:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb165d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb1d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cafb14720 con 0x563cafb0bb80 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563cafa0b470 con 0x563cafb0cd50 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563cafb148a0 con 0x563cafb0d710 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04ca23e740 10 client.0 _ll_register_callbacks cb 0x563cafa30f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59918/0 (socket says 172.21.15.32:59918) 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 -- 172.21.15.32:0/1144799750 learned_addr learned my addr 172.21.15.32:0/1144799750 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c4abd640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb165d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c5abf640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb1d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 -- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 msgr2=0x563cafb1d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb1d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 -- 172.21.15.32:0/1144799750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 msgr2=0x563cafb165d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb165d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 -- 172.21.15.32:0/1144799750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563caf84d6b0 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c5abf640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb1d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04c52be640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009c80 tx=0x7f04bc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aee271706da6bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.892+0000 7f04a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc012860 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f04bc012a00 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b801dab0 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f04bc012c00 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 10 client.15228 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563cafb209e0 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b801dab0 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f04bc01a020 con 0x563cafb0bb80 2024-04-23T18:14:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 10 client.15228 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 10 client.15228 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 20 client.15228 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb23090 0x563cafb24500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 4 client.15228 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.893+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cafa7e670 con 0x563cafb23090 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.894+0000 7f04c4abd640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb23090 0x563cafb24500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.894+0000 7f04c4abd640 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb23090 0x563cafb24500 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e2e37495fa6f85b server_cookie=76091ec8c965826c in_seq=0 out_seq=0 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.894+0000 7f04a8ff9640 10 client.15228 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04a8ff9640 1 -- 172.21.15.32:0/1144799750 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cafa7e670 con 0x563cafb23090 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04a8ff9640 10 client.15228 handle_command_reply: tid=0 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 client.15228 shutdown 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 msgr2=0x563cafb16090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009c80 tx=0x7f04bc0039c0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 shutdown_connections 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb23090 0x563cafb24500 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb1d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04a8ff9640 0 client.15228 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1144799750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb16090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 >> 172.21.15.32:0/1144799750 conn(0x563cafb07070 msgr2=0x563cafb061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 shutdown_connections 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 1 -- 172.21.15.32:0/1144799750 wait complete. 2024-04-23T18:14:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.895+0000 7f04ca23e740 20 client.15228 trim_cache size 0 max 16384 2024-04-23T18:14:58.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.913+0000 7f04ca23e740 1 Processor -- start 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.913+0000 7f04ca23e740 1 -- start start 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.913+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb1d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb0a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04ca23e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cafb22840 con 0x563cafb0bb80 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04ca23e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563cafb22f10 con 0x563cafb0cd50 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04ca23e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563cafb235e0 con 0x563cafb0d710 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59926/0 (socket says 172.21.15.32:59926) 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 -- 172.21.15.32:0/1427807623 learned_addr learned my addr 172.21.15.32:0/1427807623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c5abf640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb0a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 -- 172.21.15.32:0/1427807623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 msgr2=0x563cafb0a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c4abd640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb1d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb0a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 -- 172.21.15.32:0/1427807623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 msgr2=0x563cafb1d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb1d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 -- 172.21.15.32:0/1427807623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cafb13b80 con 0x563cafb0bb80 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c5abf640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb0a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c4abd640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb1d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04c52be640 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f04bc004060 tx=0x7f04bc004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8665a6de5dc82b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04aaffd640 1 -- 172.21.15.32:0/1427807623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc012660 con 0x563cafb0bb80 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04aaffd640 1 -- 172.21.15.32:0/1427807623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04bc012800 con 0x563cafb0bb80 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.914+0000 7f04aaffd640 1 -- 172.21.15.32:0/1427807623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc01c6a0 con 0x563cafb0bb80 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- 172.21.15.32:0/1427807623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 msgr2=0x563cafb09f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f04bc004060 tx=0x7f04bc004090 comp rx=0 tx=0).stop 2024-04-23T18:14:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- 172.21.15.32:0/1427807623 shutdown_connections 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0d710 0x563cafb0a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0cd50 0x563cafb1d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 --2- 172.21.15.32:0/1427807623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0bb80 0x563cafb09f20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- 172.21.15.32:0/1427807623 >> 172.21.15.32:0/1427807623 conn(0x563cafb07070 msgr2=0x563cafb020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- 172.21.15.32:0/1427807623 shutdown_connections 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- 172.21.15.32:0/1427807623 wait complete. 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 -- start start 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.915+0000 7f04ca23e740 1 Processor -- start 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 0x563cafb18c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb1c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cafb24380 con 0x563cafb0cd50 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563cafb23cb0 con 0x563cafb0d710 2024-04-23T18:14:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563cafb24a50 con 0x563cafb0bb80 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 10 client.0 _ll_register_callbacks cb 0x7f04bc004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59930/0 (socket says 172.21.15.32:59930) 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 -- 172.21.15.32:0/3169272697 learned_addr learned my addr 172.21.15.32:0/3169272697 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c52be640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 0x563cafb18c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c5abf640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb1c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 -- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 msgr2=0x563cafb18c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 0x563cafb18c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 -- 172.21.15.32:0/3169272697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 msgr2=0x563cafb1c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb1c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 -- 172.21.15.32:0/3169272697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cafb1c750 con 0x563cafb0cd50 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c52be640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 0x563cafb18c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c5abf640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb1c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04c4abd640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f04b0006010 tx=0x7f04b003b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77b83943e0cd66b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00f0e60 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04bc012800 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f04b00ee650 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f04b0049040 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 10 client.15234 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.916+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563cafb89040 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04bc012800 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f04b00b9390 con 0x563cafb0cd50 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 10 client.15234 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 10 client.15234 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 20 client.15234 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb8db90 0x563cafb89fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 4 client.15234 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "status"}) -- 0x563cafa784e0 con 0x563cafb8db90 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04c52be640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb8db90 0x563cafb89fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04a8ff9640 10 client.15234 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.917+0000 7f04c52be640 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb8db90 0x563cafb89fd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88ada53e5dc3174f server_cookie=a77b4ce50200222e in_seq=0 out_seq=0 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04a8ff9640 1 -- 172.21.15.32:0/3169272697 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563cafa784e0 con 0x563cafb8db90 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04a8ff9640 10 client.15234 handle_command_reply: tid=0 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 client.15234 shutdown 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 msgr2=0x563cafb19140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f04b0006010 tx=0x7f04b003b6b0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 shutdown_connections 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x563cafb8db90 0x563cafb89fd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563cafb0d710 0x563cafb1c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04a8ff9640 0 client.15234 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:14:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cafb0cd50 0x563cafb19140 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 --2- 172.21.15.32:0/3169272697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cafb0bb80 0x563cafb18c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 >> 172.21.15.32:0/3169272697 conn(0x563cafb07070 msgr2=0x563cafb05940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 shutdown_connections 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 1 -- 172.21.15.32:0/3169272697 wait complete. 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.918+0000 7f04ca23e740 20 client.15234 trim_cache size 0 max 16384 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24499, 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:58.919 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 109.880033267, 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 127.170516855 2024-04-23T18:14:58.920 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 msgr2=0x7f04ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 0x7f04ac0a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f04b8020000 tx=0x7f04b801b000 comp rx=0 tx=0).stop 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 msgr2=0x7f04c0198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04b0005940 tx=0x7f04b0031a50 comp rx=0 tx=0).stop 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 shutdown_connections 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f04ac0a3bb0 0x7f04ac0a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04c0103c90 0x7f04c019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c01032d0 0x7f04c0198770 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 --2- 172.21.15.32:0/2435995666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c01020e0 0x7f04c0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 >> 172.21.15.32:0/2435995666 conn(0x7f04c00fd650 msgr2=0x7f04c01065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 shutdown_connections 2024-04-23T18:14:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:58.919+0000 7f04c7522640 1 -- 172.21.15.32:0/2435995666 wait complete. 2024-04-23T18:14:58.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:14:59.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 -- 172.21.15.32:0/2289614040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01020e0 msgr2=0x7facf0102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 --2- 172.21.15.32:0/2289614040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01020e0 0x7facf0102500 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:59.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 -- 172.21.15.32:0/2289614040 shutdown_connections 2024-04-23T18:14:59.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 --2- 172.21.15.32:0/2289614040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf010a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 --2- 172.21.15.32:0/2289614040 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01032d0 0x7facf0103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 --2- 172.21.15.32:0/2289614040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01020e0 0x7facf0102500 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 -- 172.21.15.32:0/2289614040 >> 172.21.15.32:0/2289614040 conn(0x7facf00fd650 msgr2=0x7facf00ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 -- 172.21.15.32:0/2289614040 shutdown_connections 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.033+0000 7facf59c4640 1 -- 172.21.15.32:0/2289614040 wait complete. 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 Processor -- start 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 -- start start 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01020e0 0x7facf0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7facf010dbb0 con 0x7facf01032d0 2024-04-23T18:14:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7facf010da30 con 0x7facf0103c90 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facf59c4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7facf010dd30 con 0x7facf01020e0 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59940/0 (socket says 172.21.15.32:59940) 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 -- 172.21.15.32:0/230284211 learned_addr learned my addr 172.21.15.32:0/230284211 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7faceeffd640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01020e0 0x7facf0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facef7fe640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 -- 172.21.15.32:0/230284211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01020e0 msgr2=0x7facf0198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01020e0 0x7facf0198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 -- 172.21.15.32:0/230284211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 msgr2=0x7facf019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7face7fff640 1 -- 172.21.15.32:0/230284211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019fe80 con 0x7facf01032d0 2024-04-23T18:14:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.034+0000 7facef7fe640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7face7fff640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7facd800c970 tx=0x7facd800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7facf49c2640 1 -- 172.21.15.32:0/230284211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8007bb0 con 0x7facf01032d0 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7facf49c2640 1 -- 172.21.15.32:0/230284211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd8007d50 con 0x7facf01032d0 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7facf49c2640 1 -- 172.21.15.32:0/230284211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd800f450 con 0x7facf01032d0 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf01a0170 con 0x7facf01032d0 2024-04-23T18:14:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.035+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a0670 con 0x7facf01032d0 2024-04-23T18:14:59.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.037+0000 7facf49c2640 1 -- 172.21.15.32:0/230284211 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7facd8004f20 con 0x7facf01032d0 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.037+0000 7facf87b0740 1 Processor -- start 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.037+0000 7facf87b0740 1 -- start start 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5578d2411c50 con 0x5578d24ae710 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5578d23ae830 con 0x5578d24add50 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf87b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5578d240dc10 con 0x5578d24acb80 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59946/0 (socket says 172.21.15.32:59946) 2024-04-23T18:14:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 -- 172.21.15.32:0/1916268815 learned_addr learned my addr 172.21.15.32:0/1916268815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7faceeffd640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facef7fe640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf49c2640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 0x7facdc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 -- 172.21.15.32:0/1916268815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 msgr2=0x5578d24acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 -- 172.21.15.32:0/1916268815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 msgr2=0x5578d24ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 -- 172.21.15.32:0/1916268815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d21242d0 con 0x5578d24ae710 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7faceeffd640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facef7fe640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 0x7facdc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7face7fff640 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facd80af4a0 tx=0x7facd80b0b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cc51180e91723f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.038+0000 7facf49c2640 1 -- 172.21.15.32:0/230284211 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7facd80d01a0 con 0x7facf01032d0 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7face5ffb640 1 -- 172.21.15.32:0/1916268815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd801d070 con 0x5578d24ae710 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7face5ffb640 1 -- 172.21.15.32:0/1916268815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd80055f0 con 0x5578d24ae710 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7face5ffb640 1 -- 172.21.15.32:0/1916268815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd809cb50 con 0x5578d24ae710 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 -- 172.21.15.32:0/1916268815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 msgr2=0x5578d24b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facd80af4a0 tx=0x7facd80b0b30 comp rx=0 tx=0).stop 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facef7fe640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 0x7facdc0a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5578d24adc50 tx=0x7face8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 -- 172.21.15.32:0/1916268815 shutdown_connections 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b4fa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 --2- 172.21.15.32:0/1916268815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 -- 172.21.15.32:0/1916268815 >> 172.21.15.32:0/1916268815 conn(0x5578d24a8070 msgr2=0x5578d24a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 -- 172.21.15.32:0/1916268815 shutdown_connections 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.039+0000 7facf87b0740 1 -- 172.21.15.32:0/1916268815 wait complete. 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 -- start start 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 Processor -- start 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24ae710 0x5578d24be5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5578d24b5720 con 0x5578d24acb80 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5578d23ac470 con 0x5578d24add50 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5578d24b58a0 con 0x5578d24ae710 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facf87b0740 10 client.0 _ll_register_callbacks cb 0x5578d23d1f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7faceeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7faceeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59948/0 (socket says 172.21.15.32:59948) 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7faceeffd640 1 -- 172.21.15.32:0/1845029653 learned_addr learned my addr 172.21.15.32:0/1845029653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.040+0000 7facef7fe640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24ae710 0x5578d24be5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7face7fff640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 -- 172.21.15.32:0/1845029653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24ae710 msgr2=0x5578d24be5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24ae710 0x5578d24be5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 -- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 msgr2=0x5578d24b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 -- 172.21.15.32:0/1845029653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d21ee6b0 con 0x5578d24acb80 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7face7fff640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7faceeffd640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7facf0103130 tx=0x7face0031e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbc4788fd0a85f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0035070 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7face0004240 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5578d241f670 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7face004c070 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facf87b0740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5578d24c1990 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.041+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5578d241f670 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7face0043020 con 0x5578d24acb80 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 20 client.15246 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d24c3ff0 0x5578d24c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 4 client.15246 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d24194e0 con 0x5578d24c3ff0 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7face7fff640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d24c3ff0 0x5578d24c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7facceffd640 10 client.15246 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.042+0000 7face7fff640 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d24c3ff0 0x5578d24c5460 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc7d963f4032bb64 server_cookie=553511adc964844c in_seq=0 out_seq=0 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facceffd640 1 -- 172.21.15.32:0/1845029653 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578d24194e0 con 0x5578d24c3ff0 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facceffd640 10 client.15246 handle_command_reply: tid=0 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 client.15246 shutdown 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 msgr2=0x5578d24b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7facf0103130 tx=0x7face0031e50 comp rx=0 tx=0).stop 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 shutdown_connections 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d24c3ff0 0x5578d24c5460 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24ae710 0x5578d24be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facceffd640 0 client.15246 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 --2- 172.21.15.32:0/1845029653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24acb80 0x5578d24b7030 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 >> 172.21.15.32:0/1845029653 conn(0x5578d24a8070 msgr2=0x5578d24a71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 shutdown_connections 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.043+0000 7facf87b0740 1 -- 172.21.15.32:0/1845029653 wait complete. 2024-04-23T18:14:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.044+0000 7facf87b0740 20 client.15246 trim_cache size 0 max 16384 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 Processor -- start 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 -- start start 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24acb80 0x5578d24b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24add50 0x5578d24be570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c37a0 con 0x5578d24ae710 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c3e70 con 0x5578d24acb80 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facf87b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c4540 con 0x5578d24add50 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59962/0 (socket says 172.21.15.32:59962) 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 -- 172.21.15.32:0/2320088769 learned_addr learned my addr 172.21.15.32:0/2320088769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7faceeffd640 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24acb80 0x5578d24b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7face7fff640 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24add50 0x5578d24be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 -- 172.21.15.32:0/2320088769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24add50 msgr2=0x5578d24be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24add50 0x5578d24be570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 -- 172.21.15.32:0/2320088769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24acb80 msgr2=0x5578d24b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24acb80 0x5578d24b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.062+0000 7facef7fe640 1 -- 172.21.15.32:0/2320088769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d24ad7d0 con 0x5578d24ae710 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facef7fe640 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7face80139f0 tx=0x7face8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82425fe59e4479ff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7face57fa640 1 -- 172.21.15.32:0/2320088769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face801f950 con 0x5578d24ae710 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7face57fa640 1 -- 172.21.15.32:0/2320088769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face8017070 con 0x5578d24ae710 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7face57fa640 1 -- 172.21.15.32:0/2320088769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face801e450 con 0x5578d24ae710 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 -- 172.21.15.32:0/2320088769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 msgr2=0x5578d24b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7face80139f0 tx=0x7face8015000 comp rx=0 tx=0).stop 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 -- 172.21.15.32:0/2320088769 shutdown_connections 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24b1d70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24add50 0x5578d24be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 --2- 172.21.15.32:0/2320088769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24acb80 0x5578d24b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 -- 172.21.15.32:0/2320088769 >> 172.21.15.32:0/2320088769 conn(0x5578d24a8070 msgr2=0x5578d24a30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 -- 172.21.15.32:0/2320088769 shutdown_connections 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.063+0000 7facf87b0740 1 -- 172.21.15.32:0/2320088769 wait complete. 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 -- start start 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 Processor -- start 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24b9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c52e0 con 0x5578d24ae710 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c4c10 con 0x5578d24add50 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5578d24c59b0 con 0x5578d24acb80 2024-04-23T18:14:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facf87b0740 10 client.0 _ll_register_callbacks cb 0x7face0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59970/0 (socket says 172.21.15.32:59970) 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7facef7fe640 1 -- 172.21.15.32:0/617811907 learned_addr learned my addr 172.21.15.32:0/617811907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7face7fff640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.064+0000 7faceeffd640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 -- 172.21.15.32:0/617811907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 msgr2=0x5578d24b9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24b9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 -- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 msgr2=0x5578d24b9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 -- 172.21.15.32:0/617811907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d24bd0b0 con 0x5578d24ae710 2024-04-23T18:14:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7faceeffd640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facef7fe640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7face801caa0 tx=0x7face801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1507866aefc8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8037060 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face8017070 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7face801c650 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7face801f950 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facf87b0740 10 client.15252 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5578d24be400 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face8017070 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7face8031030 con 0x5578d24ae710 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.065+0000 7facf87b0740 10 client.15252 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facf87b0740 10 client.15252 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:14:59.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facf87b0740 20 client.15252 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d252e2a0 0x5578d252cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facf87b0740 4 client.15252 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "status"}) -- 0x5578d23959a0 con 0x5578d252e2a0 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7faceeffd640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d252e2a0 0x5578d252cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7faceeffd640 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d252e2a0 0x5578d252cfb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7eee5a27a90a5202 server_cookie=47105060e3da2e88 in_seq=0 out_seq=0 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.066+0000 7facceffd640 10 client.15252 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facceffd640 1 -- 172.21.15.32:0/617811907 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5578d23959a0 con 0x5578d252e2a0 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facceffd640 10 client.15252 handle_command_reply: tid=0 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 client.15252 shutdown 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 msgr2=0x5578d24bcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7face801caa0 tx=0x7face801cca0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 shutdown_connections 2024-04-23T18:14:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x5578d252e2a0 0x5578d252cfb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facceffd640 0 client.15252 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578d24ae710 0x5578d24bcac0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5578d24add50 0x5578d24b9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 --2- 172.21.15.32:0/617811907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5578d24acb80 0x5578d24b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 >> 172.21.15.32:0/617811907 conn(0x5578d24a8070 msgr2=0x5578d24b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 shutdown_connections 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 1 -- 172.21.15.32:0/617811907 wait complete. 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf87b0740 20 client.15252 trim_cache size 0 max 16384 2024-04-23T18:14:59.069 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24515, 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 111.038385386, 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:59.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:59.071 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 130.279454703 2024-04-23T18:14:59.071 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 msgr2=0x7facdc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 0x7facdc0a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5578d24adc50 tx=0x7face8004a40 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 msgr2=0x7facf01987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.067+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7facd800c970 tx=0x7facd800ce30 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 shutdown_connections 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7facdc0a3bb0 0x7facdc0a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf0103c90 0x7facf019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf01032d0 0x7facf01987a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 --2- 172.21.15.32:0/230284211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7facf01020e0 0x7facf0198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 >> 172.21.15.32:0/230284211 conn(0x7facf00fd650 msgr2=0x7facf01066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 shutdown_connections 2024-04-23T18:14:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.068+0000 7facf59c4640 1 -- 172.21.15.32:0/230284211 wait complete. 2024-04-23T18:14:59.073 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1163667072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:59.073 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3444833151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.073 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1790756944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- 172.21.15.32:0/3553144038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102230 msgr2=0x7efce41026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- 172.21.15.32:0/3553144038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102230 0x7efce41026b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009900 tx=0x7efcd002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- 172.21.15.32:0/3553144038 shutdown_connections 2024-04-23T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- 172.21.15.32:0/3553144038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102bf0 0x7efce4109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- 172.21.15.32:0/3553144038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102230 0x7efce41026b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- 172.21.15.32:0/3553144038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- 172.21.15.32:0/3553144038 >> 172.21.15.32:0/3553144038 conn(0x7efce40fc5f0 msgr2=0x7efce40fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- 172.21.15.32:0/3553144038 shutdown_connections 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- 172.21.15.32:0/3553144038 wait complete. 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 Processor -- start 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 -- start start 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.183+0000 7efceac10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 0x7efce4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efce410caa0 con 0x7efce4102bf0 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efce410c920 con 0x7efce4102230 2024-04-23T18:14:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efce410cc20 con 0x7efce4101040 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59982/0 (socket says 172.21.15.32:59982) 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 -- 172.21.15.32:0/3325735408 learned_addr learned my addr 172.21.15.32:0/3325735408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce3fff640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 0x7efce4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce89ac640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 -- 172.21.15.32:0/3325735408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 msgr2=0x7efce4198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 -- 172.21.15.32:0/3325735408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 msgr2=0x7efce4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 0x7efce4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 -- 172.21.15.32:0/3325735408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce419fe50 con 0x7efce4102bf0 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce3fff640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 0x7efce4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce89ac640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce91ad640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00eaa0 tx=0x7efcdc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce1ffb640 1 -- 172.21.15.32:0/3325735408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00cdb0 con 0x7efce4102bf0 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce1ffb640 1 -- 172.21.15.32:0/3325735408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcdc005020 con 0x7efce4102bf0 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce41a0140 con 0x7efce4102bf0 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efce1ffb640 1 -- 172.21.15.32:0/3325735408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc0116d0 con 0x7efce4102bf0 2024-04-23T18:14:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.184+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce41a0640 con 0x7efce4102bf0 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.186+0000 7efce1ffb640 1 -- 172.21.15.32:0/3325735408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efcdc0051d0 con 0x7efce4102bf0 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 Processor -- start 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 -- start start 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce1ffb640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 0x7efcd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d7d50 0x5636fe8d81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8defa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5636fe83bc50 con 0x5636fe8d6b80 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636fe7d8830 con 0x5636fe8d8710 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efceda01740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636fe837c10 con 0x5636fe8d7d50 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59992/0 (socket says 172.21.15.32:59992) 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce3fff640 1 -- 172.21.15.32:0/797514950 learned_addr learned my addr 172.21.15.32:0/797514950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce89ac640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 0x7efcd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce1ffb640 1 -- 172.21.15.32:0/3325735408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcdc0d06f0 con 0x7efce4102bf0 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce3fff640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d7d50 0x5636fe8d81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce91ad640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.187+0000 7efce3fff640 1 -- 172.21.15.32:0/797514950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d7d50 msgr2=0x5636fe8d81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce3fff640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d7d50 0x5636fe8d81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce3fff640 1 -- 172.21.15.32:0/797514950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 msgr2=0x5636fe8defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce3fff640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8defa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce3fff640 1 -- 172.21.15.32:0/797514950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fe54e2d0 con 0x5636fe8d6b80 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce91ad640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8defa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce89ac640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 0x7efcd40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcd80058c0 tx=0x7efcd8005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efce3fff640 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efcd0005ab0 tx=0x7efcd0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f059a33f588e928 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efcceffd640 1 -- 172.21.15.32:0/797514950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0043020 con 0x5636fe8d6b80 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efcceffd640 1 -- 172.21.15.32:0/797514950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0005040 con 0x5636fe8d6b80 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.188+0000 7efcceffd640 1 -- 172.21.15.32:0/797514950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0035050 con 0x5636fe8d6b80 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- 172.21.15.32:0/797514950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 msgr2=0x5636fe8d6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efcd0005ab0 tx=0x7efcd0009560 comp rx=0 tx=0).stop 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- 172.21.15.32:0/797514950 shutdown_connections 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8defa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d7d50 0x5636fe8d81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- 172.21.15.32:0/797514950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d6b80 0x5636fe8d6fa0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- 172.21.15.32:0/797514950 >> 172.21.15.32:0/797514950 conn(0x5636fe8d2070 msgr2=0x5636fe8d2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- 172.21.15.32:0/797514950 shutdown_connections 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- 172.21.15.32:0/797514950 wait complete. 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- start start 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 Processor -- start 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d6b80 0x5636fe8e0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d8710 0x5636fe8e8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.189+0000 7efceda01740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8df6c0 con 0x5636fe8d7d50 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636fe7d6470 con 0x5636fe8d6b80 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8df840 con 0x5636fe8d8710 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 10 client.0 _ll_register_callbacks cb 0x5636fe7fbf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60008/0 (socket says 172.21.15.32:60008) 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 -- 172.21.15.32:0/2660507133 learned_addr learned my addr 172.21.15.32:0/2660507133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce91ad640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d8710 0x5636fe8e8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce89ac640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d6b80 0x5636fe8e0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 -- 172.21.15.32:0/2660507133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d8710 msgr2=0x5636fe8e8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d8710 0x5636fe8e8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 -- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d6b80 msgr2=0x5636fe8e0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d6b80 0x5636fe8e0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 -- 172.21.15.32:0/2660507133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fe6186b0 con 0x5636fe8d7d50 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efce3fff640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efcd0005a80 tx=0x7efcd00318b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad490f50f9f79f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efcc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0043030 con 0x5636fe8d7d50 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcd00046c0 con 0x5636fe8d7d50 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd0005040 con 0x5636fe8d7d50 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efcd0035040 con 0x5636fe8d7d50 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 10 client.15264 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5636fe8eb810 con 0x5636fe8d7d50 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.190+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0005040 con 0x5636fe8d7d50 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efcd0003b70 con 0x5636fe8d7d50 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 10 client.15264 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 20 client.15264 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe8ed960 0x5636fe8eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 4 client.15264 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636fe849670 con 0x5636fe8ed960 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efce89ac640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe8ed960 0x5636fe8eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efcc3fff640 10 client.15264 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.191+0000 7efce89ac640 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe8ed960 0x5636fe8eedd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bd382c8271280c8 server_cookie=c98828636e09f679 in_seq=0 out_seq=0 2024-04-23T18:14:59.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efcc3fff640 1 -- 172.21.15.32:0/2660507133 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636fe849670 con 0x5636fe8ed960 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efcc3fff640 10 client.15264 handle_command_reply: tid=0 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 client.15264 shutdown 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 msgr2=0x5636fe8e14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efcd0005a80 tx=0x7efcd00318b0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 shutdown_connections 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe8ed960 0x5636fe8eedd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d8710 0x5636fe8e8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8e14f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efcc3fff640 0 client.15264 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 --2- 172.21.15.32:0/2660507133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d6b80 0x5636fe8e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 >> 172.21.15.32:0/2660507133 conn(0x5636fe8d2070 msgr2=0x5636fe8d11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 shutdown_connections 2024-04-23T18:14:59.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.192+0000 7efceda01740 1 -- 172.21.15.32:0/2660507133 wait complete. 2024-04-23T18:14:59.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.193+0000 7efceda01740 20 client.15264 trim_cache size 0 max 16384 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.210+0000 7efceda01740 1 Processor -- start 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.210+0000 7efceda01740 1 -- start start 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8db7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8ed110 con 0x5636fe8d7d50 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8ed7e0 con 0x5636fe8d8710 2024-04-23T18:14:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efceda01740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8edeb0 con 0x5636fe8d6b80 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60020/0 (socket says 172.21.15.32:60020) 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 -- 172.21.15.32:0/211675941 learned_addr learned my addr 172.21.15.32:0/211675941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 -- 172.21.15.32:0/211675941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 msgr2=0x5636fe8db7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8db7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 -- 172.21.15.32:0/211675941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 msgr2=0x5636fe8e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 -- 172.21.15.32:0/211675941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fe8d2d40 con 0x5636fe8d7d50 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efce3fff640 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcd0004010 tx=0x7efcd002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66119f5b85a593e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efcce7fc640 1 -- 172.21.15.32:0/211675941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0043070 con 0x5636fe8d7d50 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efcce7fc640 1 -- 172.21.15.32:0/211675941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0031dd0 con 0x5636fe8d7d50 2024-04-23T18:14:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.211+0000 7efcce7fc640 1 -- 172.21.15.32:0/211675941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd003a5f0 con 0x5636fe8d7d50 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- 172.21.15.32:0/211675941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 msgr2=0x5636fe8dbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcd0004010 tx=0x7efcd002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- 172.21.15.32:0/211675941 shutdown_connections 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d8710 0x5636fe8e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d7d50 0x5636fe8dbd30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 --2- 172.21.15.32:0/211675941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8db7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- 172.21.15.32:0/211675941 >> 172.21.15.32:0/211675941 conn(0x5636fe8d2070 msgr2=0x5636fe8d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- 172.21.15.32:0/211675941 shutdown_connections 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- 172.21.15.32:0/211675941 wait complete. 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 -- start start 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.212+0000 7efceda01740 1 Processor -- start 2024-04-23T18:14:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8e84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d7d50 0x5636fe953910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8eec50 con 0x5636fe8d8710 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8ee580 con 0x5636fe8d7d50 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636fe8ef320 con 0x5636fe8d6b80 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efceda01740 10 client.0 _ll_register_callbacks cb 0x7efcd00315e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60028/0 (socket says 172.21.15.32:60028) 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 -- 172.21.15.32:0/2979026342 learned_addr learned my addr 172.21.15.32:0/2979026342 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce3fff640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d7d50 0x5636fe953910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce89ac640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8e84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 -- 172.21.15.32:0/2979026342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 msgr2=0x5636fe8e84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8e84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 -- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d7d50 msgr2=0x5636fe953910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d7d50 0x5636fe953910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 -- 172.21.15.32:0/2979026342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fe9566d0 con 0x5636fe8d8710 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efce91ad640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0ce1c0 tx=0x7efcdc09c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7baeea5a4352ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efcc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.213+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc01f040 con 0x5636fe8d8710 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcdc0b10b0 con 0x5636fe8d8710 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd0031dd0 con 0x5636fe8d8710 2024-04-23T18:14:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efcdc09d0a0 con 0x5636fe8d8710 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 10 client.15270 fetch_fsmap learned FSMap version 21 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5636fe957aa0 con 0x5636fe8d8710 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0031dd0 con 0x5636fe8d8710 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efcdc00f040 con 0x5636fe8d8710 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 10 client.15270 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 10 client.15270 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 20 client.15270 populate_metadata read hostname 'smithi032' 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe95c630 0x5636fe959140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 4 client.15270 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "status"}) -- 0x5636fe8434e0 con 0x5636fe95c630 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.214+0000 7efce89ac640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe95c630 0x5636fe959140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efcc3fff640 10 client.15270 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efce89ac640 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe95c630 0x5636fe959140 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c023b7f1d188446 server_cookie=556655138d1ef055 in_seq=0 out_seq=0 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efcc3fff640 1 -- 172.21.15.32:0/2979026342 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5636fe8434e0 con 0x5636fe95c630 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efcc3fff640 10 client.15270 handle_command_reply: tid=0 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 client.15270 shutdown 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 msgr2=0x5636fe955ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0ce1c0 tx=0x7efcdc09c5e0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 shutdown_connections 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5636fe95c630 0x5636fe959140 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5636fe8d8710 0x5636fe955ea0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636fe8d7d50 0x5636fe953910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 --2- 172.21.15.32:0/2979026342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636fe8d6b80 0x5636fe8e84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efcc3fff640 0 client.15270 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:14:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 >> 172.21.15.32:0/2979026342 conn(0x5636fe8d2070 msgr2=0x5636fe8db7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 shutdown_connections 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 1 -- 172.21.15.32:0/2979026342 wait complete. 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.215+0000 7efceda01740 20 client.15270 trim_cache size 0 max 16384 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24551, 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:14:59.217 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 113.20948226900001, 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 21, 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 88, 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 118.363166069 2024-04-23T18:14:59.218 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 msgr2=0x7efcd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 0x7efcd40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcd80058c0 tx=0x7efcd8005d50 comp rx=0 tx=0).stop 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 msgr2=0x7efce419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00eaa0 tx=0x7efcdc00ef60 comp rx=0 tx=0).stop 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 shutdown_connections 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efce4102bf0 0x7efce419f7e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7efcd40a3bb0 0x7efcd40a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce4102230 0x7efce4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 --2- 172.21.15.32:0/3325735408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efce4101040 0x7efce4198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 >> 172.21.15.32:0/3325735408 conn(0x7efce40fc5f0 msgr2=0x7efce40fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 shutdown_connections 2024-04-23T18:14:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.216+0000 7efceac10640 1 -- 172.21.15.32:0/3325735408 wait complete. 2024-04-23T18:14:59.221 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T18:14:59.231 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T18:14:59.232 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T18:14:59.232 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T18:14:59.232 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-23T18:14:59.233 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:14:59.233 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:14:59.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-23T18:14:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3986177264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 msgr2=0x7fa3641024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3986177264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa3641024c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa354009900 tx=0x7fa35402a410 comp rx=0 tx=0).stop 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3986177264 shutdown_connections 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3986177264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364102a00 0x7fa364109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3986177264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa3641024c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3986177264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364100e60 0x7fa364101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3986177264 >> 172.21.15.32:0/3986177264 conn(0x7fa3640fc420 msgr2=0x7fa3640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3986177264 shutdown_connections 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3986177264 wait complete. 2024-04-23T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 Processor -- start 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.341+0000 7fa36a2a0640 1 -- start start 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 0x7fa36410b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 0x7fa36410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410db30 con 0x7fa364102040 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410d9b0 con 0x7fa364100e60 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410dcb0 con 0x7fa364102a00 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60046/0 (socket says 172.21.15.32:60046) 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 -- 172.21.15.32:0/3343967462 learned_addr learned my addr 172.21.15.32:0/3343967462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa363fff640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 0x7fa36410b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36883d640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 0x7fa36410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 -- 172.21.15.32:0/3343967462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 msgr2=0x7fa36410a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 0x7fa36410a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 -- 172.21.15.32:0/3343967462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 msgr2=0x7fa36410b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 0x7fa36410b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 -- 172.21.15.32:0/3343967462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36410ac70 con 0x7fa364102040 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa363fff640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 0x7fa36410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36883d640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 0x7fa36410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3637fe640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa354005940 tx=0x7fa3540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35403f030 con 0x7fa364102040 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa354003cb0 con 0x7fa364102040 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3641a22f0 con 0x7fa364102040 2024-04-23T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.342+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354035070 con 0x7fa364102040 2024-04-23T18:14:59.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.343+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3641a2840 con 0x7fa364102040 2024-04-23T18:14:59.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.344+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3641082d0 con 0x7fa364102040 2024-04-23T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.345+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa354042030 con 0x7fa364102040 2024-04-23T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.345+0000 7fa3617fa640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 0x7fa34c0a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.345+0000 7fa363fff640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 0x7fa34c0a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.345+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa3540f08e0 con 0x7fa364102040 2024-04-23T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.346+0000 7fa363fff640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 0x7fa34c0a6060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa364101bb0 tx=0x7fa358005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.349+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3540bd2f0 con 0x7fa364102040 2024-04-23T18:14:59.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1163667072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:59.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3444833151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1790756944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1163667072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:14:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3444833151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1790756944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:59.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.586+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa36402d5c0 con 0x7fa364102040 2024-04-23T18:14:59.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.975+0000 7fa3617fa640 1 -- 172.21.15.32:0/3343967462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fa354003e60 con 0x7fa364102040 2024-04-23T18:14:59.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.977+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 msgr2=0x7fa34c0a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.977+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 0x7fa34c0a6060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa364101bb0 tx=0x7fa358005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.977+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 msgr2=0x7fa364109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.977+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa354005940 tx=0x7fa3540042f0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 shutdown_connections 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364102a00 0x7fa36410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102040 0x7fa364109ea0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fa34c0a3ba0 0x7fa34c0a6060 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 --2- 172.21.15.32:0/3343967462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364100e60 0x7fa36410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 >> 172.21.15.32:0/3343967462 conn(0x7fa3640fc420 msgr2=0x7fa3641054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 shutdown_connections 2024-04-23T18:14:59.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:14:59.978+0000 7fa36a2a0640 1 -- 172.21.15.32:0/3343967462 wait complete. 2024-04-23T18:14:59.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.105+0000 7f6b79631640 1 -- 172.21.15.32:0/2914318739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 msgr2=0x7f6b7410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.105+0000 7f6b79631640 1 --2- 172.21.15.32:0/2914318739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7410a0e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009ca0 tx=0x7f6b6c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 -- 172.21.15.32:0/2914318739 shutdown_connections 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 --2- 172.21.15.32:0/2914318739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7410a0e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 --2- 172.21.15.32:0/2914318739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74105360 0x7f6b74107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 --2- 172.21.15.32:0/2914318739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74102a30 0x7f6b74104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 -- 172.21.15.32:0/2914318739 >> 172.21.15.32:0/2914318739 conn(0x7f6b740fc420 msgr2=0x7f6b740fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 -- 172.21.15.32:0/2914318739 shutdown_connections 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 -- 172.21.15.32:0/2914318739 wait complete. 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 Processor -- start 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.106+0000 7f6b79631640 1 -- start start 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74102a30 0x7f6b74198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74105360 0x7f6b741986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7410db40 con 0x7f6b74107c90 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7410d9c0 con 0x7f6b74102a30 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7410dcc0 con 0x7f6b74105360 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60076/0 (socket says 172.21.15.32:60076) 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b72ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74102a30 0x7f6b74198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 -- 172.21.15.32:0/440373148 learned_addr learned my addr 172.21.15.32:0/440373148 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b727fc640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74105360 0x7f6b741986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74105360 msgr2=0x7f6b741986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74105360 0x7f6b741986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74102a30 msgr2=0x7f6b74198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74102a30 0x7f6b74198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 -- 172.21.15.32:0/440373148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7419fd90 con 0x7f6b74107c90 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b737fe640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c033040 tx=0x7f6b6c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0317c0 con 0x7f6b74107c90 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b741a0020 con 0x7f6b74107c90 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b6c0044f0 con 0x7f6b74107c90 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c02fd00 con 0x7f6b74107c90 2024-04-23T18:15:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.107+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b741a0570 con 0x7f6b74107c90 2024-04-23T18:15:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.108+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b740fdf10 con 0x7f6b74107c90 2024-04-23T18:15:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.110+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f6b6c004040 con 0x7f6b74107c90 2024-04-23T18:15:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.110+0000 7f6b5bfff640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 0x7f6b5c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.110+0000 7f6b72ffd640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 msgr2=0x7f6b5c0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.110+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6b6c0f1b40 con 0x7f6b74107c90 2024-04-23T18:15:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.110+0000 7f6b72ffd640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 0x7f6b5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:15:00.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.118+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b6c0be550 con 0x7f6b74107c90 2024-04-23T18:15:00.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:59 smithi032 ceph-mon[29221]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:15:00.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:14:59 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:15:00.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setuser ceph since I am not root 2024-04-23T18:15:00.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: ignoring --setgroup ceph since I am not root 2024-04-23T18:15:00.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:00.172+0000 7f45ec487200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:15:00.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.307+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f6b7402d5c0 con 0x7f6b74107c90 2024-04-23T18:15:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.311+0000 7f6b72ffd640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 msgr2=0x7f6b5c0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.311+0000 7f6b72ffd640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 0x7f6b5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:15:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.311+0000 7f6b5bfff640 1 -- 172.21.15.32:0/440373148 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f6b6c0bde50 con 0x7f6b74107c90 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 msgr2=0x7f6b5c0a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 0x7f6b5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 msgr2=0x7f6b7419f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c033040 tx=0x7f6b6c003ba0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 shutdown_connections 2024-04-23T18:15:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b74107c90 0x7f6b7419f720 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b74105360 0x7f6b741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f6b5c0a3bf0 0x7f6b5c0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 --2- 172.21.15.32:0/440373148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b74102a30 0x7f6b74198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 >> 172.21.15.32:0/440373148 conn(0x7f6b740fc420 msgr2=0x7f6b74103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 shutdown_connections 2024-04-23T18:15:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.313+0000 7f6b79631640 1 -- 172.21.15.32:0/440373148 wait complete. 2024-04-23T18:15:00.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-23T18:15:00.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:59 smithi064 ceph-mon[33062]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:15:00.348 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:14:59 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:15:00.349 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: ignoring --setuser ceph since I am not root 2024-04-23T18:15:00.349 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: ignoring --setgroup ceph since I am not root 2024-04-23T18:15:00.349 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-mgr[34925]: -- 172.21.15.64:0/3347853783 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fdae6225a0 con 0x55fdae5f6c00 2024-04-23T18:15:00.349 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:00.164+0000 7ff78da69200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:15:00.349 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:00.354+0000 7ff78da69200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:00.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:59 smithi040 ceph-mon[32626]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:15:00.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:14:59 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:15:00.351 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: ignoring --setuser ceph since I am not root 2024-04-23T18:15:00.351 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: ignoring --setgroup ceph since I am not root 2024-04-23T18:15:00.351 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-mgr[34016]: -- 172.21.15.40:0/3786048523 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f517ee5a0 con 0x558f517c2800 2024-04-23T18:15:00.351 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:00.165+0000 7fdce716d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:15:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1607558389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0101fd0 msgr2=0x7f8be01023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1607558389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0101fd0 0x7f8be01023f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009900 tx=0x7f8bd002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1607558389 shutdown_connections 2024-04-23T18:15:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1607558389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be0103b80 0x7f8be010a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1607558389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be01031c0 0x7f8be0103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1607558389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0101fd0 0x7f8be01023f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1607558389 >> 172.21.15.32:0/1607558389 conn(0x7f8be00fd540 msgr2=0x7f8be00ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1607558389 shutdown_connections 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.429+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1607558389 wait complete. 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 Processor -- start 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 -- start start 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be0101fd0 0x7f8be0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be01031c0 0x7f8be0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8be010db60 con 0x7f8be0103b80 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8be010d9e0 con 0x7f8be0101fd0 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8be5c7c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8be010dce0 con 0x7f8be01031c0 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60088/0 (socket says 172.21.15.32:60088) 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdffff640 1 -- 172.21.15.32:0/1909540420 learned_addr learned my addr 172.21.15.32:0/1909540420 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdeffd640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be01031c0 0x7f8be0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdf7fe640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be0101fd0 0x7f8be0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.430+0000 7f8bdffff640 1 -- 172.21.15.32:0/1909540420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be01031c0 msgr2=0x7f8be0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdffff640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be01031c0 0x7f8be0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdffff640 1 -- 172.21.15.32:0/1909540420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be0101fd0 msgr2=0x7f8be0198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdffff640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be0101fd0 0x7f8be0198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdffff640 1 -- 172.21.15.32:0/1909540420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be019fe30 con 0x7f8be0103b80 2024-04-23T18:15:00.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdffff640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800eb10 tx=0x7f8bd800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8004b20 con 0x7f8be0103b80 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be01a0120 con 0x7f8be0103b80 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bd8004cc0 con 0x7f8be0103b80 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8010690 con 0x7f8be0103b80 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.431+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be01a06c0 con 0x7f8be0103b80 2024-04-23T18:15:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.432+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be0109410 con 0x7f8be0103b80 2024-04-23T18:15:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.433+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f8bd8005000 con 0x7f8be0103b80 2024-04-23T18:15:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.434+0000 7f8bdcff9640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 0x7f8bc80a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.434+0000 7f8bdf7fe640 1 -- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 msgr2=0x7f8bc80a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.434+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8bd80d0bf0 con 0x7f8be0103b80 2024-04-23T18:15:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.434+0000 7f8bdf7fe640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 0x7f8bc80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:15:00.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.438+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bd8053ff0 con 0x7f8be0103b80 2024-04-23T18:15:00.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:00.356+0000 7fdce716d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:00.617 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:00.462+0000 7fdce716d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:15:00.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.626+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f8be0062bc0 con 0x7f8be0103b80 2024-04-23T18:15:00.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.630+0000 7f8bdcff9640 1 -- 172.21.15.32:0/1909540420 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7f8bd809d5a0 con 0x7f8be0103b80 2024-04-23T18:15:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 msgr2=0x7f8bc80a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:15:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 0x7f8bc80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 msgr2=0x7f8be019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800eb10 tx=0x7f8bd800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 shutdown_connections 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8be0103b80 0x7f8be019f7c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8be01031c0 0x7f8be0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f8bc80a3bf0 0x7f8bc80a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 --2- 172.21.15.32:0/1909540420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be0101fd0 0x7f8be0198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 >> 172.21.15.32:0/1909540420 conn(0x7f8be00fd540 msgr2=0x7f8be01065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 shutdown_connections 2024-04-23T18:15:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:00.632+0000 7f8be5c7c640 1 -- 172.21.15.32:0/1909540420 wait complete. 2024-04-23T18:15:00.636 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:00.367+0000 7f45ec487200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:00.636 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:00.479+0000 7f45ec487200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:15:00.638 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T18:15:00.663 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T18:15:00.664 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T18:15:00.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:00.664 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-23T18:15:00.671 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:00.460+0000 7ff78da69200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:15:00.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:00.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-23T18:15:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-04-23T18:15:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:15:00.749 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-23T18:15:00.749 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T18:15:00.750 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-23T18:15:00.750 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-23T18:15:00.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-23T18:15:00.890 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:00.806+0000 7f45ec487200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:15:00.891 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:00.896+0000 7f45ec487200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:15:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-mon[32626]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/440373148' entity='client.admin' 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1909540420' entity='client.admin' 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:00.782+0000 7fdce716d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:15:01.117 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:00.870+0000 7fdce716d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:15:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:15:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-mon[29221]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-23T18:15:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/440373148' entity='client.admin' 2024-04-23T18:15:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1909540420' entity='client.admin' 2024-04-23T18:15:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3343967462' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:15:01.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-mon[33062]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-23T18:15:01.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/440373148' entity='client.admin' 2024-04-23T18:15:01.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1909540420' entity='client.admin' 2024-04-23T18:15:01.172 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:00.779+0000 7ff78da69200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:15:01.172 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:00.866+0000 7ff78da69200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:15:02.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:02.110+0000 7fdce716d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:15:02.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:02.287+0000 7fdce716d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:02.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:02.105+0000 7ff78da69200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:15:02.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:02.281+0000 7ff78da69200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:02.491 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:02.174+0000 7f45ec487200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:15:02.491 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:02.367+0000 7f45ec487200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:02.750 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:02.457+0000 7ff78da69200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:15:02.756 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:02.462+0000 7fdce716d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:15:02.757 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:02.548+0000 7f45ec487200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:15:02.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 -- 172.21.15.32:0/3276603899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a8107980 msgr2=0x7f73a8107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 --2- 172.21.15.32:0/3276603899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a8107980 0x7f73a8107d60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f739c009990 tx=0x7f739c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 -- 172.21.15.32:0/3276603899 shutdown_connections 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 --2- 172.21.15.32:0/3276603899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a81022f0 0x7f73a810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 --2- 172.21.15.32:0/3276603899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a8107980 0x7f73a8107d60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 --2- 172.21.15.32:0/3276603899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8101930 0x7f73a8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 -- 172.21.15.32:0/3276603899 >> 172.21.15.32:0/3276603899 conn(0x7f73a80788b0 msgr2=0x7f73a8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 -- 172.21.15.32:0/3276603899 shutdown_connections 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.899+0000 7f73aec8a700 1 -- 172.21.15.32:0/3276603899 wait complete. 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 Processor -- start 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 -- start start 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a8101930 0x7f73a8199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8107980 0x7f73a81a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73a8111b20 con 0x7f73a81022f0 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73a81119a0 con 0x7f73a8101930 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aec8a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f73a8111ca0 con 0x7f73a8107980 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60114/0 (socket says 172.21.15.32:60114) 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73a7fff700 1 -- 172.21.15.32:0/753355458 learned_addr learned my addr 172.21.15.32:0/753355458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.900+0000 7f73aca26700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a8101930 0x7f73a8199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73ad227700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8107980 0x7f73a81a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8107980 msgr2=0x7f73a81a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8107980 0x7f73a81a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a8101930 msgr2=0x7f73a8199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a8101930 0x7f73a8199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 -- 172.21.15.32:0/753355458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a81a5560 con 0x7f73a81022f0 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a7fff700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f739400d980 tx=0x7f739400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394016070 con 0x7f73a81022f0 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7394004830 con 0x7f73a81022f0 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a81a5790 con 0x7f73a81022f0 2024-04-23T18:15:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394005540 con 0x7f73a81022f0 2024-04-23T18:15:02.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.901+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a81a5ca0 con 0x7f73a81022f0 2024-04-23T18:15:02.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.902+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a8109ea0 con 0x7f73a81022f0 2024-04-23T18:15:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.903+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f7394004220 con 0x7f73a81022f0 2024-04-23T18:15:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.904+0000 7f73a5ffb700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 0x7f73980a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.904+0000 7f73aca26700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 msgr2=0x7f73980a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.904+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f73940d01d0 con 0x7f73a81022f0 2024-04-23T18:15:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.904+0000 7f73aca26700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 0x7f73980a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:15:02.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:02.909+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f739401f020 con 0x7f73a81022f0 2024-04-23T18:15:03.056 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:02.756+0000 7ff78da69200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:15:03.056 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:02.851+0000 7ff78da69200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:15:03.056 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:03.062+0000 7ff78da69200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:15:03.061 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:02.762+0000 7fdce716d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:15:03.061 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:02.857+0000 7fdce716d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:15:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.105+0000 7f73aca26700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 msgr2=0x7f73980a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.105+0000 7f73aca26700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 0x7f73980a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:15:03.099 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:02.870+0000 7f45ec487200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:15:03.100 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:02.971+0000 7f45ec487200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:15:03.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.168+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f73a8069730 con 0x7f73a81022f0 2024-04-23T18:15:03.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.169+0000 7f73a5ffb700 1 -- 172.21.15.32:0/753355458 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f739409cbe0 con 0x7f73a81022f0 2024-04-23T18:15:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-23T18:15:03.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 msgr2=0x7f73980a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 0x7f73980a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 msgr2=0x7f73a819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f739400d980 tx=0x7f739400dc90 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 shutdown_connections 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73a8107980 0x7f73a81a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f73980a3c90 0x7f73980a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a8101930 0x7f73a8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 --2- 172.21.15.32:0/753355458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73a81022f0 0x7f73a819a4e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 >> 172.21.15.32:0/753355458 conn(0x7f73a80788b0 msgr2=0x7f73a81006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 shutdown_connections 2024-04-23T18:15:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03.171+0000 7f73aec8a700 1 -- 172.21.15.32:0/753355458 wait complete. 2024-04-23T18:15:03.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/753355458' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:15:03.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:03.067+0000 7fdce716d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:15:03.367 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:03.267+0000 7fdce716d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:03.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/753355458' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:15:03.386 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:03.186+0000 7f45ec487200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:15:03.386 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:03.392+0000 7f45ec487200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:03.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/753355458' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:15:03.422 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:03.262+0000 7ff78da69200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:15:03.856 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T18:15:03.856 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T18:15:03.856 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-04-23T18:15:03.856 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T18:15:03.857 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-23T18:15:03.857 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T18:15:03.857 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T18:15:03.857 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T18:15:03.857 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-23T18:15:03.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:03.857 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-04-23T18:15:03.874 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41972sec preferred_lft 41972sec 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:15:03.875 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-04-23T18:15:03.876 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T18:15:03.876 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-23T18:15:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:04.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:04.064 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T18:15:04.077 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:04.030+0000 7f45ec487200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:15:04.089 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-04-23T18:15:04.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:04.091 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-04-23T18:15:04.160 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-04-23T18:15:04.160 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T18:15:04.160 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-23T18:15:04.160 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-23T18:15:04.160 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-23T18:15:04.161 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-23T18:15:04.171 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:03.884+0000 7ff78da69200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:15:04.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:04.322 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:03.889+0000 7fdce716d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:15:04.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:04.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:04.419 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-23T18:15:04.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:04.435 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-04-23T18:15:04.491 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:15:04.491 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-23T18:15:04.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-23T18:15:04.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-23T18:15:04.491 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-23T18:15:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:04.588 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:04.499+0000 7f45ec487200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:15:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:04.614 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-23T18:15:04.614 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:15:04.614 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-23T18:15:04.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-23T18:15:04.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T18:15:04.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-23T18:15:04.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-23T18:15:04.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-23T18:15:04.615 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-23T18:15:04.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:04.328+0000 7fdce716d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:15:04.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:04.427+0000 7fdce716d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:15:04.624 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:04.322+0000 7ff78da69200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:15:04.624 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:04.422+0000 7ff78da69200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:15:04.624 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:04.630+0000 7ff78da69200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:15:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:04.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:04.749 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:15:04.749 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-23T18:15:04.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-04-23T18:15:04.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T18:15:04.749 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-23T18:15:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:04.830 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:15:04.831 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /tmp/cephfs 2024-04-23T18:15:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-23T18:15:04.847 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:15:04.847 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /tmp/cephfs 2024-04-23T18:15:04.903 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-23T18:15:04.903 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-23T18:15:04.903 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/d17f95c9-36ff-4b3f-b9ea-0f18c979081d 2024-04-23T18:15:04.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:04.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/d17f95c9-36ff-4b3f-b9ea-0f18c979081d /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0 2024-04-23T18:15:04.965 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:04.608+0000 7f45ec487200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:15:04.965 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:04.836+0000 7f45ec487200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:15:05.008 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-23T18:15:05.020 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-23T18:15:05.021 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-04-23T18:15:05.116 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:04.636+0000 7fdce716d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:15:05.120 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-23T18:15:05.120 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-23T18:15:05.121 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-04-23T18:15:05.121 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.40:3300,172.21.15.64:3300" 2024-04-23T18:15:05.121 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:3300,172.21.15.40:3300,172.21.15.64:3300:/volumes/qa/sv_1/d17f95c9-36ff-4b3f-b9ea-0f18c979081d 2024-04-23T18:15:05.121 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=cdb712f8-019b-11ef-bc93-c7b262605968" will pass to kernel 2024-04-23T18:15:05.123 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T18:15:05.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:15:05.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /tmp/cephfs 2024-04-23T18:15:05.193 DEBUG:teuthology.orchestra.run.smithi032:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:15:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:15:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:05.293 INFO:teuthology.orchestra.run.smithi032.stdout:client25003 2024-04-23T18:15:05.293 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:15:05.294 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:15:05.310 DEBUG:teuthology.orchestra.run.smithi032:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:15:05.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:15:05.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:client25003 2024-04-23T18:15:05.410 DEBUG:teuthology.orchestra.run.smithi032:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:15:05.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:15:05.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:05.469 INFO:teuthology.orchestra.run.smithi032.stdout:client25003 2024-04-23T18:15:05.470 DEBUG:teuthology.orchestra.run.smithi032:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/cdb712f8-019b-11ef-bc93-c7b262605968.client25003/status') 2024-04-23T18:15:05.520 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.179+0000 7ff78da69200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:15:05.521 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.274+0000 7ff78da69200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:15:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:15:05.524 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.181+0000 7fdce716d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:15:05.524 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.278+0000 7fdce716d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:15:05.531 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:05.401+0000 7f45ec487200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:15:05.532 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:05.504+0000 7f45ec487200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:15:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.25003 (3)172.21.15.32:0/1358759481 2024-04-23T18:15:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-04-23T18:15:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-04-23T18:15:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-04-23T18:15:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:71 bytes copied, 2.4744e-05 s, 2.9 MB/s 2024-04-23T18:15:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:15:05.543 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:15:05.555 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:15:05.555 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:15:05.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2539726255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce18107cd0 msgr2=0x7fce1810a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2539726255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce18107cd0 0x7fce1810a120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fce10009ca0 tx=0x7fce1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2539726255 shutdown_connections 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2539726255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce18107cd0 0x7fce1810a120 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2539726255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce181053a0 0x7fce18107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2539726255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 0x7fce18104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2539726255 >> 172.21.15.32:0/2539726255 conn(0x7fce180fc480 msgr2=0x7fce180fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.668+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2539726255 shutdown_connections 2024-04-23T18:15:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.669+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2539726255 wait complete. 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.669+0000 7fce1f7f0640 1 Processor -- start 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.669+0000 7fce1f7f0640 1 -- start start 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 0x7fce181981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce18107cd0 0x7fce1819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810db80 con 0x7fce181053a0 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810da00 con 0x7fce18102a70 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1f7f0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810dd00 con 0x7fce18107cd0 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60120/0 (socket says 172.21.15.32:60120) 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 -- 172.21.15.32:0/2437884924 learned_addr learned my addr 172.21.15.32:0/2437884924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1dd8d640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce18107cd0 0x7fce1819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1d58c640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 0x7fce181981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce18107cd0 msgr2=0x7fce1819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce18107cd0 0x7fce1819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 msgr2=0x7fce181981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 0x7fce181981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 -- 172.21.15.32:0/2437884924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1819fde0 con 0x7fce181053a0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.670+0000 7fce1cd8b640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e9f0 tx=0x7fce0800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.671+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0800ccc0 con 0x7fce181053a0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.671+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce080054d0 con 0x7fce181053a0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.671+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181a00d0 con 0x7fce181053a0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.671+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08010430 con 0x7fce181053a0 2024-04-23T18:15:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.671+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce181a05d0 con 0x7fce181053a0 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.672+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce180fdf70 con 0x7fce181053a0 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.673+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fce080046c0 con 0x7fce181053a0 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.674+0000 7fce0e7fc640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 0x7fce040a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.674+0000 7fce1d58c640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 msgr2=0x7fce040a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.674+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fce080d0990 con 0x7fce181053a0 2024-04-23T18:15:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.674+0000 7fce1d58c640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 0x7fce040a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:15:05.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.678+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce08053ff0 con 0x7fce181053a0 2024-04-23T18:15:05.796 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.526+0000 7ff78da69200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:15:05.796 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.672+0000 7ff78da69200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:15:05.796 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.772+0000 7ff78da69200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:15:05.866 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.530+0000 7fdce716d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:15:05.867 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.671+0000 7fdce716d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:15:05.867 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.769+0000 7fdce716d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:15:05.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.875+0000 7fce1d58c640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 msgr2=0x7fce040a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:05.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.875+0000 7fce1d58c640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 0x7fce040a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:15:05.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.922+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fce1802d5c0 con 0x7fce181053a0 2024-04-23T18:15:05.924 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:05.783+0000 7f45ec487200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:15:05.924 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:05.930+0000 7f45ec487200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:15:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.974+0000 7fce0e7fc640 1 -- 172.21.15.32:0/2437884924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v22) ==== 117+0+0 (secure 0 0 0) 0x7fce0809d3a0 con 0x7fce181053a0 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 msgr2=0x7fce040a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 0x7fce040a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 msgr2=0x7fce18198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e9f0 tx=0x7fce0800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 shutdown_connections 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce18107cd0 0x7fce1819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce181053a0 0x7fce18198700 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7fce040a3bf0 0x7fce040a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 --2- 172.21.15.32:0/2437884924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce18102a70 0x7fce181981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 >> 172.21.15.32:0/2437884924 conn(0x7fce180fc480 msgr2=0x7fce181033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:05.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 shutdown_connections 2024-04-23T18:15:05.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:05.976+0000 7fce1f7f0640 1 -- 172.21.15.32:0/2437884924 wait complete. 2024-04-23T18:15:05.983 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-23T18:15:05.998 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:15:06.054 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:15:06.054 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:15:06.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-23T18:15:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.168+0000 7f0e52493640 1 -- 172.21.15.32:0/4291074140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c102040 msgr2=0x7f0e4c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.168+0000 7f0e52493640 1 --2- 172.21.15.32:0/4291074140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c102040 0x7f0e4c1024c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009900 tx=0x7f0e4002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- 172.21.15.32:0/4291074140 shutdown_connections 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- 172.21.15.32:0/4291074140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e4c102a00 0x7f0e4c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- 172.21.15.32:0/4291074140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c102040 0x7f0e4c1024c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- 172.21.15.32:0/4291074140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e4c100e60 0x7f0e4c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- 172.21.15.32:0/4291074140 >> 172.21.15.32:0/4291074140 conn(0x7f0e4c0fc420 msgr2=0x7f0e4c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- 172.21.15.32:0/4291074140 shutdown_connections 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- 172.21.15.32:0/4291074140 wait complete. 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 Processor -- start 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- start start 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 0x7f0e4c104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e4c102040 0x7f0e4c1034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e4c102a00 0x7f0e4c1039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c10ca00 con 0x7f0e4c100e60 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c10c880 con 0x7f0e4c102040 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.169+0000 7f0e52493640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c10cb80 con 0x7f0e4c102a00 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 0x7f0e4c104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 0x7f0e4c104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60128/0 (socket says 172.21.15.32:60128) 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 -- 172.21.15.32:0/2351610205 learned_addr learned my addr 172.21.15.32:0/2351610205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e4c102040 0x7f0e4c1034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e50a30640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e4c102a00 0x7f0e4c1039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e4c102a00 msgr2=0x7f0e4c1039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e4c102a00 0x7f0e4c1039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e4c102040 msgr2=0x7f0e4c1034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e4c102040 0x7f0e4c1034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c1042b0 con 0x7f0e4c100e60 2024-04-23T18:15:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e4bfff640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 0x7f0e4c104e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00e9c0 tx=0x7f0e3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00cd80 con 0x7f0e4c100e60 2024-04-23T18:15:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e4c1a23a0 con 0x7f0e4c100e60 2024-04-23T18:15:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e3c0035f0 con 0x7f0e4c100e60 2024-04-23T18:15:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c010690 con 0x7f0e4c100e60 2024-04-23T18:15:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.170+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4c1a28c0 con 0x7f0e4c100e60 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.172+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e4c1a2b50 con 0x7f0e4c100e60 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.173+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0e3c0037a0 con 0x7f0e4c100e60 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.174+0000 7f0e497fa640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 0x7f0e340a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.174+0000 7f0e4b7fe640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 msgr2=0x7f0e340a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.174+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0e3c0d0770 con 0x7f0e4c100e60 2024-04-23T18:15:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.174+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 0x7f0e340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:15:06.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.178+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e3c09d180 con 0x7f0e4c100e60 2024-04-23T18:15:06.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:15:06.172 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:05.897+0000 7ff78da69200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:06.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:15:06.195 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:05.889+0000 7fdce716d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:06.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:15:06.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:06.037+0000 7f45ec487200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:15:06.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:06.165+0000 7f45ec487200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:15:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.375+0000 7f0e4b7fe640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 msgr2=0x7f0e340a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.375+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 0x7f0e340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:15:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.391+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0e4c069e80 con 0x7f0e340a3bf0 2024-04-23T18:15:06.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:06.201+0000 7fdce716d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:15:06.672 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:06.218+0000 7ff78da69200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:15:06.757 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:06.488+0000 7f45ec487200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:15:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.776+0000 7f0e4b7fe640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 msgr2=0x7f0e340a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3652066649 2024-04-23T18:15:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:06.776+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 0x7f0e340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:15:07.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:06 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:15:07.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:06 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:07.070 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:06.851+0000 7ff78da69200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:15:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:06 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:15:07.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:06 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:07.117 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:06.828+0000 7fdce716d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:15:07.117 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y[34012]: 2024-04-23T18:15:07.051+0000 7fdce716d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:15:07.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:06 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2437884924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:15:07.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:06 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:07.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.128+0000 7f45ec487200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:15:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:07.370+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 20) ==== 138331+0+0 (secure 0 0 0) 0x7f0e3c0990f0 con 0x7f0e4c100e60 2024-04-23T18:15:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:07.371+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 msgr2=0x7f0e340a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:15:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:07.371+0000 7f0e497fa640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3652066649,v1:172.21.15.32:6801/3652066649] conn(0x7f0e340a3bf0 0x7f0e340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.421 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z[34921]: 2024-04-23T18:15:07.075+0000 7ff78da69200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:15:07.606 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.359+0000 7f45ec487200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:15:07.975 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.611+0000 7f454bdbb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.975 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.611+0000 7f454bdbb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.975 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.611+0000 7f454bdbb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.975 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.611+0000 7f454bdbb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.611+0000 7f454bdbb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.612+0000 7f454e5c0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.612+0000 7f454e5c0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.612+0000 7f454e5c0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.612+0000 7f454e5c0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:07.976 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:15:07.612+0000 7f454e5c0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Standby manager daemon y restarted 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Standby manager daemon y started 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Standby manager daemon z restarted 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Standby manager daemon z started 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Active manager daemon x restarted 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Activating manager daemon x 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: mgrmap e20: x(active, starting, since 0.0119861s), standbys: z, y 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:15:08.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:08.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:08.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: Manager daemon x is now available 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:15:08.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:07 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Standby manager daemon y restarted 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Standby manager daemon y started 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Standby manager daemon z restarted 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Standby manager daemon z started 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Active manager daemon x restarted 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Activating manager daemon x 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: mgrmap e20: x(active, starting, since 0.0119861s), standbys: z, y 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:15:08.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:15:08.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:08.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: Manager daemon x is now available 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:15:08.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:07 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:15:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.377+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7f0e3c0987d0 con 0x7f0e4c100e60 2024-04-23T18:15:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.378+0000 7f0e497fa640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0e34052120 0x7f0e34052500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.378+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0e3c015e80 con 0x7f0e34052120 2024-04-23T18:15:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.383+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0e34052120 0x7f0e34052500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.386+0000 7f0e4b7fe640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0e34052120 0x7f0e34052500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000c00 tx=0x7f0e400314a0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Standby manager daemon y restarted 2024-04-23T18:15:08.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Standby manager daemon y started 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.40:0/1671187702' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Standby manager daemon z restarted 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Standby manager daemon z started 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.? 172.21.15.64:0/1786585970' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Active manager daemon x restarted 2024-04-23T18:15:08.423 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Activating manager daemon x 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: mgrmap e20: x(active, starting, since 0.0119861s), standbys: z, y 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:15:08.424 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:15:08.425 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: Manager daemon x is now available 2024-04-23T18:15:08.426 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:08.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:15:08.427 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:07 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:15:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.458+0000 7f0e497fa640 1 -- 172.21.15.32:0/2351610205 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0e3c015e80 con 0x7f0e34052120 2024-04-23T18:15:08.453 INFO:teuthology.orchestra.run.smithi032.stdout:Schedule set for path / 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0e34052120 msgr2=0x7f0e34052500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0e34052120 0x7f0e34052500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000c00 tx=0x7f0e400314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 msgr2=0x7f0e4c104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 --2- 172.21.15.32:0/2351610205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e4c100e60 0x7f0e4c104e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00e9c0 tx=0x7f0e3c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e50a30640 1 -- 172.21.15.32:0/2351610205 reap_dead start 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 shutdown_connections 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 >> 172.21.15.32:0/2351610205 conn(0x7f0e4c0fc420 msgr2=0x7f0e4c069770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 shutdown_connections 2024-04-23T18:15:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.461+0000 7f0e52493640 1 -- 172.21.15.32:0/2351610205 wait complete. 2024-04-23T18:15:08.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-23T18:15:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2267552760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101200 msgr2=0x7ff634101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2267552760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101200 0x7ff634101660 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff630009900 tx=0x7ff63002a480 comp rx=0 tx=0).stop 2024-04-23T18:15:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2267552760 shutdown_connections 2024-04-23T18:15:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2267552760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff634101ba0 0x7ff634102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2267552760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101200 0x7ff634101660 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2267552760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6341008a0 0x7ff634100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2267552760 >> 172.21.15.32:0/2267552760 conn(0x7ff6340fc420 msgr2=0x7ff6340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2267552760 shutdown_connections 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.578+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2267552760 wait complete. 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 Processor -- start 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 -- start start 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6341008a0 0x7ff63419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 0x7ff63419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff63410fcf0 con 0x7ff634101ba0 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff63410fb70 con 0x7ff634101200 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63bf9b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff63410fe70 con 0x7ff6341008a0 2024-04-23T18:15:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40768/0 (socket says 172.21.15.32:40768) 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 -- 172.21.15.32:0/2701499285 learned_addr learned my addr 172.21.15.32:0/2701499285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff639536640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 0x7ff63419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff639d37640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6341008a0 0x7ff63419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 -- 172.21.15.32:0/2701499285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6341008a0 msgr2=0x7ff63419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6341008a0 0x7ff63419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 -- 172.21.15.32:0/2701499285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 msgr2=0x7ff63419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 0x7ff63419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff63a538640 1 -- 172.21.15.32:0/2701499285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6341a2020 con 0x7ff634101ba0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.579+0000 7ff639536640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 0x7ff63419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff63a538640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00eaa0 tx=0x7ff62c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c00cdb0 con 0x7ff634101ba0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff62c005020 con 0x7ff634101ba0 2024-04-23T18:15:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6341a2310 con 0x7ff634101ba0 2024-04-23T18:15:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0116d0 con 0x7ff634101ba0 2024-04-23T18:15:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.580+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6341a2830 con 0x7ff634101ba0 2024-04-23T18:15:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.581+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63410b860 con 0x7ff634101ba0 2024-04-23T18:15:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.582+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7ff62c0051d0 con 0x7ff634101ba0 2024-04-23T18:15:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.583+0000 7ff62affd640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 0x7ff6200a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.583+0000 7ff639d37640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 0x7ff6200a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.583+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff62c0d01e0 con 0x7ff634101ba0 2024-04-23T18:15:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.583+0000 7ff639d37640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 0x7ff6200a5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff624009bd0 tx=0x7ff624005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.587+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff62c016030 con 0x7ff634101ba0 2024-04-23T18:15:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.801+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ff634062430 con 0x7ff6200a3990 2024-04-23T18:15:08.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.856+0000 7ff62affd640 1 -- 172.21.15.32:0/2701499285 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ff634062430 con 0x7ff6200a3990 2024-04-23T18:15:08.850 INFO:teuthology.orchestra.run.smithi032.stdout:Retention added to path / 2024-04-23T18:15:08.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 msgr2=0x7ff6200a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 0x7ff6200a5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff624009bd0 tx=0x7ff624005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 msgr2=0x7ff6341a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00eaa0 tx=0x7ff62c00ef60 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 shutdown_connections 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6200a3990 0x7ff6200a5e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff634101ba0 0x7ff6341a19b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff634101200 0x7ff63419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 --2- 172.21.15.32:0/2701499285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6341008a0 0x7ff63419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 >> 172.21.15.32:0/2701499285 conn(0x7ff6340fc420 msgr2=0x7ff6340fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 shutdown_connections 2024-04-23T18:15:08.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.858+0000 7ff63bf9b640 1 -- 172.21.15.32:0/2701499285 wait complete. 2024-04-23T18:15:08.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:15:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 -- 172.21.15.32:0/1737035367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7260103b90 msgr2=0x7f726010a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 --2- 172.21.15.32:0/1737035367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7260103b90 0x7f726010a420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7258009ca0 tx=0x7f725802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 -- 172.21.15.32:0/1737035367 shutdown_connections 2024-04-23T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 --2- 172.21.15.32:0/1737035367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7260103b90 0x7f726010a420 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 --2- 172.21.15.32:0/1737035367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72601031d0 0x7f7260103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 --2- 172.21.15.32:0/1737035367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 0x7f7260102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 -- 172.21.15.32:0/1737035367 >> 172.21.15.32:0/1737035367 conn(0x7f72600fd570 msgr2=0x7f72600ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 -- 172.21.15.32:0/1737035367 shutdown_connections 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.977+0000 7f726627c640 1 -- 172.21.15.32:0/1737035367 wait complete. 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 Processor -- start 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 -- start start 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 0x7f7260198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7260103b90 0x7f726019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f726010dae0 con 0x7f72601031d0 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f726010d960 con 0x7f7260101fe0 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f726627c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f726010dc60 con 0x7f7260103b90 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f725f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f725f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40788/0 (socket says 172.21.15.32:40788) 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f725f7fe640 1 -- 172.21.15.32:0/3987803129 learned_addr learned my addr 172.21.15.32:0/3987803129 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f725ffff640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 0x7f7260198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.978+0000 7f7264819640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7260103b90 0x7f726019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 -- 172.21.15.32:0/3987803129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7260103b90 msgr2=0x7f726019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7260103b90 0x7f726019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 -- 172.21.15.32:0/3987803129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 msgr2=0x7f7260198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 0x7f7260198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 -- 172.21.15.32:0/3987803129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726019fda0 con 0x7f72601031d0 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725f7fe640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f725000e9f0 tx=0x7f725000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725000cd80 con 0x7f72601031d0 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72500108e0 con 0x7f72601031d0 2024-04-23T18:15:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250018980 con 0x7f72601031d0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72601a0090 con 0x7f72601031d0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.979+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72601a05e0 con 0x7f72601031d0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.980+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7260109380 con 0x7f72601031d0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.981+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7f7250010430 con 0x7f72601031d0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.982+0000 7f725d7fa640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 0x7f724c0a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.982+0000 7f725ffff640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 0x7f724c0a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.982+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f72500d03e0 con 0x7f72601031d0 2024-04-23T18:15:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.982+0000 7f725ffff640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 0x7f724c0a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7260103030 tx=0x7f7248005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:08.987+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f725009c950 con 0x7f72601031d0 2024-04-23T18:15:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.211+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7260062390 con 0x7f724c0a3990 2024-04-23T18:15:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.220+0000 7f725d7fa640 1 -- 172.21.15.32:0/3987803129 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f7260062390 con 0x7f724c0a3990 2024-04-23T18:15:09.214 INFO:teuthology.orchestra.run.smithi032.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:15:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 msgr2=0x7f724c0a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 0x7f724c0a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7260103030 tx=0x7f7248005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 msgr2=0x7f72601986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f725000e9f0 tx=0x7f725000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 shutdown_connections 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7260103b90 0x7f726019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f724c0a3990 0x7f724c0a5e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72601031d0 0x7f72601986c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 --2- 172.21.15.32:0/3987803129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7260101fe0 0x7f7260198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 >> 172.21.15.32:0/3987803129 conn(0x7f72600fd570 msgr2=0x7f7260106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 shutdown_connections 2024-04-23T18:15:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.222+0000 7f726627c640 1 -- 172.21.15.32:0/3987803129 wait complete. 2024-04-23T18:15:09.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:15:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 -- 172.21.15.32:0/2458727941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 msgr2=0x7f6c64101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 --2- 172.21.15.32:0/2458727941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64101280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009900 tx=0x7f6c5402a410 comp rx=0 tx=0).stop 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 -- 172.21.15.32:0/2458727941 shutdown_connections 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 --2- 172.21.15.32:0/2458727941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102a00 0x7f6c64109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 --2- 172.21.15.32:0/2458727941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102040 0x7f6c641024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 --2- 172.21.15.32:0/2458727941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64101280 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 -- 172.21.15.32:0/2458727941 >> 172.21.15.32:0/2458727941 conn(0x7f6c640fc420 msgr2=0x7f6c640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 -- 172.21.15.32:0/2458727941 shutdown_connections 2024-04-23T18:15:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.349+0000 7f6c69cae640 1 -- 172.21.15.32:0/2458727941 wait complete. 2024-04-23T18:15:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.350+0000 7f6c69cae640 1 Processor -- start 2024-04-23T18:15:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.350+0000 7f6c69cae640 1 -- start start 2024-04-23T18:15:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.350+0000 7f6c69cae640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.350+0000 7f6c69cae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102040 0x7f6c641976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.350+0000 7f6c69cae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102a00 0x7f6c6419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c69cae640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6410c9f0 con 0x7f6c64100e60 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c69cae640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6410c870 con 0x7f6c64102a00 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c69cae640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6410cb70 con 0x7f6c64102040 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40804/0 (socket says 172.21.15.32:40804) 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 -- 172.21.15.32:0/3779567776 learned_addr learned my addr 172.21.15.32:0/3779567776 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c62ffd640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102040 0x7f6c641976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c63fff640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102a00 0x7f6c6419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 -- 172.21.15.32:0/3779567776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102040 msgr2=0x7f6c641976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102040 0x7f6c641976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 -- 172.21.15.32:0/3779567776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102a00 msgr2=0x7f6c6419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102a00 0x7f6c6419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 -- 172.21.15.32:0/3779567776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6419ed50 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c637fe640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5403f030 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6419efe0 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c54031910 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.351+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54035070 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.352+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c6419f440 con 0x7f6c64100e60 2024-04-23T18:15:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.353+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c64108310 con 0x7f6c64100e60 2024-04-23T18:15:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.354+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7f6c54042030 con 0x7f6c64100e60 2024-04-23T18:15:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.355+0000 7f6c60ff9640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 0x7f6c4c0a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.355+0000 7f6c62ffd640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 0x7f6c4c0a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.355+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f6c540f0590 con 0x7f6c64100e60 2024-04-23T18:15:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.356+0000 7f6c62ffd640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 0x7f6c4c0a5e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6c64198640 tx=0x7f6c58005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.360+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c54003e60 con 0x7f6c64100e60 2024-04-23T18:15:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.384+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7f6c540b91b0 con 0x7f6c64100e60 2024-04-23T18:15:09.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: mgrmap e21: x(active, since 1.02034s), standbys: z, y 2024-04-23T18:15:09.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: [23/Apr/2024:18:15:08] ENGINE Bus STARTING 2024-04-23T18:15:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: [23/Apr/2024:18:15:08] ENGINE Serving on https://172.21.15.32:7150 2024-04-23T18:15:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: [23/Apr/2024:18:15:08] ENGINE Serving on http://172.21.15.32:8765 2024-04-23T18:15:09.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:09 smithi032 ceph-mon[29221]: [23/Apr/2024:18:15:08] ENGINE Bus STARTED 2024-04-23T18:15:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.605+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f6c64062390 con 0x7f6c4c0a3990 2024-04-23T18:15:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.613+0000 7f6c60ff9640 1 -- 172.21.15.32:0/3779567776 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f6c64062390 con 0x7f6c4c0a3990 2024-04-23T18:15:09.606 INFO:teuthology.orchestra.run.smithi032.stdout:/ 1m 6m3h 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 msgr2=0x7f6c4c0a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 0x7f6c4c0a5e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6c64198640 tx=0x7f6c58005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 msgr2=0x7f6c64197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c540042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 shutdown_connections 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c64102a00 0x7f6c6419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c64102040 0x7f6c641976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6c4c0a3990 0x7f6c4c0a5e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 --2- 172.21.15.32:0/3779567776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c64100e60 0x7f6c64197180 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 >> 172.21.15.32:0/3779567776 conn(0x7f6c640fc420 msgr2=0x7f6c641054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 shutdown_connections 2024-04-23T18:15:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.616+0000 7f6c69cae640 1 -- 172.21.15.32:0/3779567776 wait complete. 2024-04-23T18:15:09.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: mgrmap e21: x(active, since 1.02034s), standbys: z, y 2024-04-23T18:15:09.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: [23/Apr/2024:18:15:08] ENGINE Bus STARTING 2024-04-23T18:15:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: [23/Apr/2024:18:15:08] ENGINE Serving on https://172.21.15.32:7150 2024-04-23T18:15:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: [23/Apr/2024:18:15:08] ENGINE Serving on http://172.21.15.32:8765 2024-04-23T18:15:09.617 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:09 smithi040 ceph-mon[32626]: [23/Apr/2024:18:15:08] ENGINE Bus STARTED 2024-04-23T18:15:09.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-23T18:15:09.651 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-23T18:15:09.661 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-23T18:15:09.727 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:15:09.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-23T18:15:09.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: mgrmap e21: x(active, since 1.02034s), standbys: z, y 2024-04-23T18:15:09.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: [23/Apr/2024:18:15:08] ENGINE Bus STARTING 2024-04-23T18:15:09.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: [23/Apr/2024:18:15:08] ENGINE Serving on https://172.21.15.32:7150 2024-04-23T18:15:09.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:09.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: [23/Apr/2024:18:15:08] ENGINE Serving on http://172.21.15.32:8765 2024-04-23T18:15:09.729 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:09 smithi064 ceph-mon[33062]: [23/Apr/2024:18:15:08] ENGINE Bus STARTED 2024-04-23T18:15:09.731 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-23T18:15:09.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:09.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 -- 172.21.15.32:0/4238584866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 msgr2=0x7f6bd41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/4238584866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41024c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009900 tx=0x7f6bbc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 -- 172.21.15.32:0/4238584866 shutdown_connections 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/4238584866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 0x7f6bd4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/4238584866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41024c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/4238584866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 0x7f6bd4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 -- 172.21.15.32:0/4238584866 >> 172.21.15.32:0/4238584866 conn(0x7f6bd40fc420 msgr2=0x7f6bd40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 -- 172.21.15.32:0/4238584866 shutdown_connections 2024-04-23T18:15:09.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.855+0000 7f6bda1ed640 1 -- 172.21.15.32:0/4238584866 wait complete. 2024-04-23T18:15:09.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 Processor -- start 2024-04-23T18:15:09.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 -- start start 2024-04-23T18:15:09.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 0x7f6bd4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 0x7f6bd419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd410db30 con 0x7f6bd4102040 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd410d9b0 con 0x7f6bd4100e60 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.856+0000 7f6bda1ed640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd410dcb0 con 0x7f6bd4102a00 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40834/0 (socket says 172.21.15.32:40834) 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 0x7f6bd4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 -- 172.21.15.32:0/1297812204 learned_addr learned my addr 172.21.15.32:0/1297812204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bd3fff640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 0x7f6bd419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 -- 172.21.15.32:0/1297812204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 msgr2=0x7f6bd419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 0x7f6bd419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 -- 172.21.15.32:0/1297812204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 msgr2=0x7f6bd4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 0x7f6bd4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 -- 172.21.15.32:0/1297812204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd419fda0 con 0x7f6bd4102040 2024-04-23T18:15:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bcaffd640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc005940 tx=0x7f6bbc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc03f030 con 0x7f6bd4102040 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd41a0030 con 0x7f6bd4102040 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bbc003aa0 con 0x7f6bd4102040 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.857+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc035070 con 0x7f6bd4102040 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.858+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd41a0520 con 0x7f6bd4102040 2024-04-23T18:15:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.859+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd4108250 con 0x7f6bd4102040 2024-04-23T18:15:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.860+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7f6bbc042030 con 0x7f6bd4102040 2024-04-23T18:15:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.860+0000 7f6bd17fa640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 0x7f6bc00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.860+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f6bbc0f0d40 con 0x7f6bd4102040 2024-04-23T18:15:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.861+0000 7f6bd37fe640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 0x7f6bc00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.861+0000 7f6bd37fe640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 0x7f6bc00a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4005fd0 tx=0x7f6bc400b410 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:09.865+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bbc0bcde0 con 0x7f6bd4102040 2024-04-23T18:15:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.057+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bd402d7e0 con 0x7f6bd4102040 2024-04-23T18:15:10.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.058+0000 7f6bd17fa640 1 -- 172.21.15.32:0/1297812204 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22910 (secure 0 0 0) 0x7f6bbc0c10e0 con 0x7f6bd4102040 2024-04-23T18:15:10.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:10.053 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":89,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:15:07.361332+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:10.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 msgr2=0x7f6bc00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 0x7f6bc00a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4005fd0 tx=0x7f6bc400b410 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 msgr2=0x7f6bd41986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc005940 tx=0x7f6bbc031a50 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 shutdown_connections 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bd4102a00 0x7f6bd419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6bc00a3d20 0x7f6bc00a61e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bd4102040 0x7f6bd41986c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 --2- 172.21.15.32:0/1297812204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bd4100e60 0x7f6bd4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.060+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 >> 172.21.15.32:0/1297812204 conn(0x7f6bd40fc420 msgr2=0x7f6bd4105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.061+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 shutdown_connections 2024-04-23T18:15:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.061+0000 7f6bda1ed640 1 -- 172.21.15.32:0/1297812204 wait complete. 2024-04-23T18:15:10.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:10.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:15:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 -- 172.21.15.32:0/2728018534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 msgr2=0x7efed4101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 --2- 172.21.15.32:0/2728018534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed4101fe0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7efecc009ca0 tx=0x7efecc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 -- 172.21.15.32:0/2728018534 shutdown_connections 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 --2- 172.21.15.32:0/2728018534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed4101fe0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 --2- 172.21.15.32:0/2728018534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 --2- 172.21.15.32:0/2728018534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 -- 172.21.15.32:0/2728018534 >> 172.21.15.32:0/2728018534 conn(0x7efed40fc420 msgr2=0x7efed40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 -- 172.21.15.32:0/2728018534 shutdown_connections 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.182+0000 7efeda6fa640 1 -- 172.21.15.32:0/2728018534 wait complete. 2024-04-23T18:15:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.183+0000 7efeda6fa640 1 Processor -- start 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.183+0000 7efeda6fa640 1 -- start start 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efed410fd30 con 0x7efed4101b60 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efed410fbb0 con 0x7efed41011c0 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efeda6fa640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efed410feb0 con 0x7efed4100860 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40848/0 (socket says 172.21.15.32:40848) 2024-04-23T18:15:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 -- 172.21.15.32:0/1350171854 learned_addr learned my addr 172.21.15.32:0/1350171854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed37fe640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 -- 172.21.15.32:0/1350171854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 msgr2=0x7efed419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 -- 172.21.15.32:0/1350171854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 msgr2=0x7efed419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed419a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 -- 172.21.15.32:0/1350171854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed41a1f80 con 0x7efed4101b60 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed3fff640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed37fe640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.184+0000 7efed8c97640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efecc033040 tx=0x7efecc0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.185+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc003b70 con 0x7efed4101b60 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.185+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efecc003d10 con 0x7efed4101b60 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.185+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc02fb40 con 0x7efed4101b60 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.185+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed41a2210 con 0x7efed4101b60 2024-04-23T18:15:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.185+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed41a2710 con 0x7efed4101b60 2024-04-23T18:15:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.186+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed410b5b0 con 0x7efed4101b60 2024-04-23T18:15:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.187+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7efecc004610 con 0x7efed4101b60 2024-04-23T18:15:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.188+0000 7efed17fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.188+0000 7efed3fff640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.188+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7efecc0f1a40 con 0x7efed4101b60 2024-04-23T18:15:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.189+0000 7efed3fff640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 0x7efec00a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efec4009fc0 tx=0x7efec4005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.193+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efecc0be240 con 0x7efed4101b60 2024-04-23T18:15:10.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.382+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efed4062b50 con 0x7efed4101b60 2024-04-23T18:15:10.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.383+0000 7efed17fa640 1 -- 172.21.15.32:0/1350171854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7efecc0bdb40 con 0x7efed4101b60 2024-04-23T18:15:10.377 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 msgr2=0x7efec00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 0x7efec00a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efec4009fc0 tx=0x7efec4005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 msgr2=0x7efed41a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efecc033040 tx=0x7efecc0316f0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 shutdown_connections 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed4101b60 0x7efed41a1910 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed41011c0 0x7efed419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 --2- 172.21.15.32:0/1350171854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efed4100860 0x7efed419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 >> 172.21.15.32:0/1350171854 conn(0x7efed40fc420 msgr2=0x7efed40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 shutdown_connections 2024-04-23T18:15:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.386+0000 7efeda6fa640 1 -- 172.21.15.32:0/1350171854 wait complete. 2024-04-23T18:15:10.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1329178081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568103250 msgr2=0x7fc5681036d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1329178081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568103250 0x7fc5681036d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc550009930 tx=0x7fc55002a420 comp rx=0 tx=0).stop 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1329178081 shutdown_connections 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1329178081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 0x7fc56810a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1329178081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568103250 0x7fc5681036d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.507+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1329178081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568100280 0x7fc5681006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1329178081 >> 172.21.15.32:0/1329178081 conn(0x7fc5680fbbb0 msgr2=0x7fc5680fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1329178081 shutdown_connections 2024-04-23T18:15:10.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1329178081 wait complete. 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 Processor -- start 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- start start 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568103250 0x7fc568101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 0x7fc568101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810db20 con 0x7fc568100280 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810d9a0 con 0x7fc568103250 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc56d2ef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810dca0 con 0x7fc568103c10 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.508+0000 7fc566ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40874/0 (socket says 172.21.15.32:40874) 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 -- 172.21.15.32:0/1368742360 learned_addr learned my addr 172.21.15.32:0/1368742360 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc5677fe640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 0x7fc568101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc55ffff640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568103250 0x7fc568101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 -- 172.21.15.32:0/1368742360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 msgr2=0x7fc568101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 0x7fc568101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 -- 172.21.15.32:0/1368742360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568103250 msgr2=0x7fc568101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568103250 0x7fc568101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 -- 172.21.15.32:0/1368742360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc568102000 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc566ffd640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc55800e9c0 tx=0x7fc55800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55800cd80 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5681a2370 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5580035f0 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558010690 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.509+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5681a28e0 con 0x7fc568100280 2024-04-23T18:15:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.510+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5681094a0 con 0x7fc568100280 2024-04-23T18:15:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.512+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7fc5580037a0 con 0x7fc568100280 2024-04-23T18:15:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.513+0000 7fc564ff9640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 0x7fc5540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.513+0000 7fc55ffff640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 0x7fc5540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.513+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc5580d0af0 con 0x7fc568100280 2024-04-23T18:15:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.514+0000 7fc55ffff640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 0x7fc5540a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc550006010 tx=0x7fc5500314a0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.518+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55809d1e0 con 0x7fc568100280 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.648 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.649 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:10 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1297812204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:10.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:10 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1297812204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.709+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc568062b50 con 0x7fc568100280 2024-04-23T18:15:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.709+0000 7fc564ff9640 1 -- 172.21.15.32:0/1368742360 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fc55809cae0 con 0x7fc568100280 2024-04-23T18:15:10.704 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 msgr2=0x7fc5540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 0x7fc5540a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc550006010 tx=0x7fc5500314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 msgr2=0x7fc568102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc55800e9c0 tx=0x7fc55800ee80 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 shutdown_connections 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.711+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc568103c10 0x7fc568101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc568103250 0x7fc568101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc5540a3d20 0x7fc5540a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 --2- 172.21.15.32:0/1368742360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc568100280 0x7fc568102b70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 >> 172.21.15.32:0/1368742360 conn(0x7fc5680fbbb0 msgr2=0x7fc568106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 shutdown_connections 2024-04-23T18:15:10.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.712+0000 7fc56d2ef640 1 -- 172.21.15.32:0/1368742360 wait complete. 2024-04-23T18:15:10.715 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:10.716 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:10 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1297812204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:10.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 -- 172.21.15.32:0/3511606744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 msgr2=0x7f2cc81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 --2- 172.21.15.32:0/3511606744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc81012b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009900 tx=0x7f2cbc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 -- 172.21.15.32:0/3511606744 shutdown_connections 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 --2- 172.21.15.32:0/3511606744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 --2- 172.21.15.32:0/3511606744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 --2- 172.21.15.32:0/3511606744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc81012b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 -- 172.21.15.32:0/3511606744 >> 172.21.15.32:0/3511606744 conn(0x7f2cc80fc420 msgr2=0x7f2cc80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 -- 172.21.15.32:0/3511606744 shutdown_connections 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.838+0000 7f2ccd120640 1 -- 172.21.15.32:0/3511606744 wait complete. 2024-04-23T18:15:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.839+0000 7f2ccd120640 1 Processor -- start 2024-04-23T18:15:10.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.839+0000 7f2ccd120640 1 -- start start 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc81987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810db30 con 0x7f2cc8100e90 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810d9b0 con 0x7f2cc8101fe0 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2ccd120640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810dcb0 con 0x7f2cc81029a0 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40894/0 (socket says 172.21.15.32:40894) 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc81987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 -- 172.21.15.32:0/4199523670 learned_addr learned my addr 172.21.15.32:0/4199523670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc759e640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 -- 172.21.15.32:0/4199523670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 msgr2=0x7f2cc819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 -- 172.21.15.32:0/4199523670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 msgr2=0x7f2cc81987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc81987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 -- 172.21.15.32:0/4199523670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc819fe60 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc759e640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc659c640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.840+0000 7f2cc6d9d640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc005940 tx=0x7f2cbc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.841+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc03f030 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.841+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc81a00f0 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.841+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cbc003cb0 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.841+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc81a0630 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.841+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc035070 con 0x7f2cc8100e90 2024-04-23T18:15:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.842+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc8108330 con 0x7f2cc8100e90 2024-04-23T18:15:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.843+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7f2cbc042030 con 0x7f2cc8100e90 2024-04-23T18:15:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.844+0000 7f2caffff640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 0x7f2cb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.844+0000 7f2cc659c640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 0x7f2cb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.844+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f2cbc0f0bd0 con 0x7f2cc8100e90 2024-04-23T18:15:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.844+0000 7f2cc659c640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 0x7f2cb00a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8199750 tx=0x7f2cb8005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:10.848+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cbc0bd260 con 0x7f2cc8100e90 2024-04-23T18:15:11.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.063+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2cc8062b50 con 0x7f2cc8100e90 2024-04-23T18:15:11.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.063+0000 7f2caffff640 1 -- 172.21.15.32:0/4199523670 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f2cbc003e60 con 0x7f2cc8100e90 2024-04-23T18:15:11.057 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:15:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 msgr2=0x7f2cb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 0x7f2cb00a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8199750 tx=0x7f2cb8005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 msgr2=0x7f2cc8198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc005940 tx=0x7f2cbc0042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 shutdown_connections 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cc81029a0 0x7f2cc819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cc8101fe0 0x7f2cc81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2cb00a3d20 0x7f2cb00a61e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 --2- 172.21.15.32:0/4199523670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cc8100e90 0x7f2cc8198290 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 >> 172.21.15.32:0/4199523670 conn(0x7f2cc80fc420 msgr2=0x7f2cc8105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.065+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 shutdown_connections 2024-04-23T18:15:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.066+0000 7f2ccd120640 1 -- 172.21.15.32:0/4199523670 wait complete. 2024-04-23T18:15:11.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 -- 172.21.15.32:0/876011333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 msgr2=0x7fad2810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 --2- 172.21.15.32:0/876011333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad2810a400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad20009ca0 tx=0x7fad2002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 -- 172.21.15.32:0/876011333 shutdown_connections 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 --2- 172.21.15.32:0/876011333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad2810a400 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 --2- 172.21.15.32:0/876011333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad281031b0 0x7fad28103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 --2- 172.21.15.32:0/876011333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad28101fc0 0x7fad281023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 -- 172.21.15.32:0/876011333 >> 172.21.15.32:0/876011333 conn(0x7fad280fd530 msgr2=0x7fad280ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 -- 172.21.15.32:0/876011333 shutdown_connections 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.183+0000 7fad2f20b640 1 -- 172.21.15.32:0/876011333 wait complete. 2024-04-23T18:15:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 Processor -- start 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 -- start start 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad281031b0 0x7fad28074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad2810db50 con 0x7fad28103b70 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad2810d9d0 con 0x7fad28101fc0 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2f20b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fad2810dcd0 con 0x7fad281031b0 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2cfa7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40904/0 (socket says 172.21.15.32:40904) 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 -- 172.21.15.32:0/3640191705 learned_addr learned my addr 172.21.15.32:0/3640191705 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2cfa7640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:44804/0 (socket says 172.21.15.32:44804) 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad27fff640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad281031b0 0x7fad28074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 -- 172.21.15.32:0/3640191705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad281031b0 msgr2=0x7fad28074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad281031b0 0x7fad28074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 -- 172.21.15.32:0/3640191705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 msgr2=0x7fad28075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2d7a8640 1 -- 172.21.15.32:0/3640191705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad28075110 con 0x7fad28103b70 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.184+0000 7fad2cfa7640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad2d7a8640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad20033040 tx=0x7fad200317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20003b70 con 0x7fad28103b70 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad281a22c0 con 0x7fad28103b70 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad20003d10 con 0x7fad28103b70 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2002fbd0 con 0x7fad28103b70 2024-04-23T18:15:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.185+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad281a2810 con 0x7fad28103b70 2024-04-23T18:15:11.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.186+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad281092e0 con 0x7fad28103b70 2024-04-23T18:15:11.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.187+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7fad2002fd70 con 0x7fad28103b70 2024-04-23T18:15:11.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.188+0000 7fad25ffb640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 0x7fad180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.188+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fad200f1ae0 con 0x7fad28103b70 2024-04-23T18:15:11.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.188+0000 7fad2cfa7640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 0x7fad180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.189+0000 7fad2cfa7640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 0x7fad180a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad28103010 tx=0x7fad1c005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.193+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad200be2e0 con 0x7fad28103b70 2024-04-23T18:15:11.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.396+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad28062b50 con 0x7fad28103b70 2024-04-23T18:15:11.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.398+0000 7fad25ffb640 1 -- 172.21.15.32:0/3640191705 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22910 (secure 0 0 0) 0x7fad200bdbe0 con 0x7fad28103b70 2024-04-23T18:15:11.392 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:11.392 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":89,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:15:07.361332+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 msgr2=0x7fad180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 0x7fad180a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad28103010 tx=0x7fad1c005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 msgr2=0x7fad28074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad20033040 tx=0x7fad200317c0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 shutdown_connections 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fad180a3d20 0x7fad180a61e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad28103b70 0x7fad28074880 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad281031b0 0x7fad28074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 --2- 172.21.15.32:0/3640191705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad28101fc0 0x7fad28075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 >> 172.21.15.32:0/3640191705 conn(0x7fad280fd530 msgr2=0x7fad281064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 shutdown_connections 2024-04-23T18:15:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.400+0000 7fad2f20b640 1 -- 172.21.15.32:0/3640191705 wait complete. 2024-04-23T18:15:11.406 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:11.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:15:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 -- 172.21.15.32:0/2541514702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c108150 msgr2=0x7fd99c108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 --2- 172.21.15.32:0/2541514702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c108150 0x7fd99c108530 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009930 tx=0x7fd98c02a4d0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 -- 172.21.15.32:0/2541514702 shutdown_connections 2024-04-23T18:15:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 --2- 172.21.15.32:0/2541514702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c1011d0 0x7fd99c10edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 --2- 172.21.15.32:0/2541514702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c100810 0x7fd99c100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 --2- 172.21.15.32:0/2541514702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c108150 0x7fd99c108530 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 -- 172.21.15.32:0/2541514702 >> 172.21.15.32:0/2541514702 conn(0x7fd99c076170 msgr2=0x7fd99c076580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 -- 172.21.15.32:0/2541514702 shutdown_connections 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 -- 172.21.15.32:0/2541514702 wait complete. 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.520+0000 7fd9a160b640 1 Processor -- start 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 -- start start 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c1011d0 0x7fd99c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c108150 0x7fd99c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c112220 con 0x7fd99c100810 2024-04-23T18:15:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1120a0 con 0x7fd99c1011d0 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd9a160b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1123a0 con 0x7fd99c108150 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40930/0 (socket says 172.21.15.32:40930) 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 -- 172.21.15.32:0/4153327933 learned_addr learned my addr 172.21.15.32:0/4153327933 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99a7fc640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c1011d0 0x7fd99c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99b7fe640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c108150 0x7fd99c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 -- 172.21.15.32:0/4153327933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c108150 msgr2=0x7fd99c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c108150 0x7fd99c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 -- 172.21.15.32:0/4153327933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c1011d0 msgr2=0x7fd99c198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c1011d0 0x7fd99c198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 -- 172.21.15.32:0/4153327933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c19d750 con 0x7fd99c100810 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.521+0000 7fd99affd640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009230 tx=0x7fd98c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.522+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c03f030 con 0x7fd99c100810 2024-04-23T18:15:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.522+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98c004440 con 0x7fd99c100810 2024-04-23T18:15:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.522+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c19d9e0 con 0x7fd99c100810 2024-04-23T18:15:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.522+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c035070 con 0x7fd99c100810 2024-04-23T18:15:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.522+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a52d0 con 0x7fd99c100810 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.523+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c069250 con 0x7fd99c100810 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.524+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 138766+0+0 (secure 0 0 0) 0x7fd98c042030 con 0x7fd99c100810 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.524+0000 7fd983fff640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 0x7fd9840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.524+0000 7fd99a7fc640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 0x7fd9840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.524+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fd98c0f0a70 con 0x7fd99c100810 2024-04-23T18:15:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.525+0000 7fd99a7fc640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 0x7fd9840a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd99c107730 tx=0x7fd990005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.529+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd98c0bcb10 con 0x7fd99c100810 2024-04-23T18:15:11.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1350171854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:15:11.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1368742360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-23T18:15:11.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-23T18:15:11.675 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:11.675 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:11 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4199523670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.723+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd99c10a490 con 0x7fd99c100810 2024-04-23T18:15:11.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.724+0000 7fd983fff640 1 -- 172.21.15.32:0/4153327933 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fd98c0c0e10 con 0x7fd99c100810 2024-04-23T18:15:11.718 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:15:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 msgr2=0x7fd9840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 0x7fd9840a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd99c107730 tx=0x7fd990005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 msgr2=0x7fd99c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009230 tx=0x7fd98c004040 comp rx=0 tx=0).stop 2024-04-23T18:15:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 shutdown_connections 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c108150 0x7fd99c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c1011d0 0x7fd99c198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9840a3d20 0x7fd9840a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 --2- 172.21.15.32:0/4153327933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd99c100810 0x7fd99c198760 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 >> 172.21.15.32:0/4153327933 conn(0x7fd99c076170 msgr2=0x7fd99c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 shutdown_connections 2024-04-23T18:15:11.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.726+0000 7fd9a160b640 1 -- 172.21.15.32:0/4153327933 wait complete. 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1350171854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.732 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1368742360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:15:11.733 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:11.734 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:11 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4199523670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- 172.21.15.32:0/3340470575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c10c590 msgr2=0x7f3e4c10e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 --2- 172.21.15.32:0/3340470575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c10c590 0x7f3e4c10e9e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009be0 tx=0x7f3e4402a750 comp rx=0 tx=0).stop 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- 172.21.15.32:0/3340470575 shutdown_connections 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 --2- 172.21.15.32:0/3340470575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c10c590 0x7f3e4c10e9e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 --2- 172.21.15.32:0/3340470575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c103ae0 0x7f3e4c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 --2- 172.21.15.32:0/3340470575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- 172.21.15.32:0/3340470575 >> 172.21.15.32:0/3340470575 conn(0x7f3e4c077fa0 msgr2=0x7f3e4c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- 172.21.15.32:0/3340470575 shutdown_connections 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- 172.21.15.32:0/3340470575 wait complete. 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 Processor -- start 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.854+0000 7f3e5164e640 1 -- start start 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c103ae0 0x7f3e4c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c10c590 0x7f3e4c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1121a0 con 0x7f3e4c1031c0 2024-04-23T18:15:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c112020 con 0x7f3e4c10c590 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c112320 con 0x7f3e4c103ae0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40936/0 (socket says 172.21.15.32:40936) 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 -- 172.21.15.32:0/1180793208 learned_addr learned my addr 172.21.15.32:0/1180793208 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4b7fe640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c10c590 0x7f3e4c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4a7fc640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c103ae0 0x7f3e4c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 -- 172.21.15.32:0/1180793208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c103ae0 msgr2=0x7f3e4c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c103ae0 0x7f3e4c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 -- 172.21.15.32:0/1180793208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c10c590 msgr2=0x7f3e4c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c10c590 0x7f3e4c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 -- 172.21.15.32:0/1180793208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c19d690 con 0x7f3e4c1031c0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e4affd640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00e9c0 tx=0x7f3e3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00ccc0 con 0x7f3e4c1031c0 2024-04-23T18:15:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.855+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c19d980 con 0x7f3e4c1031c0 2024-04-23T18:15:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.856+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e3c0054d0 con 0x7f3e4c1031c0 2024-04-23T18:15:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.856+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c010430 con 0x7f3e4c1031c0 2024-04-23T18:15:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.856+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c1a5210 con 0x7f3e4c1031c0 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.856+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4c06ae80 con 0x7f3e4c1031c0 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.858+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f3e3c004f80 con 0x7f3e4c1031c0 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.858+0000 7f3e33fff640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 0x7f3e340a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.859+0000 7f3e4a7fc640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 0x7f3e340a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.859+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3e3c0d0dd0 con 0x7f3e4c1031c0 2024-04-23T18:15:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.859+0000 7f3e4a7fc640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 0x7f3e340a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c10e2d0 tx=0x7f3e40005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:11.863+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e3c09d350 con 0x7f3e4c1031c0 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1350171854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1368742360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:11.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:11.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:11 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4199523670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.062+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3e4c1999f0 con 0x7f3e4c1031c0 2024-04-23T18:15:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.062+0000 7f3e33fff640 1 -- 172.21.15.32:0/1180793208 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f3e3c09cc50 con 0x7f3e4c1031c0 2024-04-23T18:15:12.056 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 msgr2=0x7f3e340a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 0x7f3e340a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c10e2d0 tx=0x7f3e40005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 msgr2=0x7f3e4c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00e9c0 tx=0x7f3e3c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 shutdown_connections 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e4c10c590 0x7f3e4c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e4c103ae0 0x7f3e4c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3e340a3db0 0x7f3e340a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 --2- 172.21.15.32:0/1180793208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4c1031c0 0x7f3e4c1986a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 >> 172.21.15.32:0/1180793208 conn(0x7f3e4c077fa0 msgr2=0x7f3e4c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 shutdown_connections 2024-04-23T18:15:12.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.064+0000 7f3e5164e640 1 -- 172.21.15.32:0/1180793208 wait complete. 2024-04-23T18:15:12.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:15:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 -- 172.21.15.32:0/87314737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101a40 msgr2=0x7fac4c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 --2- 172.21.15.32:0/87314737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101a40 0x7fac4c105a80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fac40009900 tx=0x7fac4002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 -- 172.21.15.32:0/87314737 shutdown_connections 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 --2- 172.21.15.32:0/87314737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c106160 0x7fac4c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 --2- 172.21.15.32:0/87314737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101a40 0x7fac4c105a80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 --2- 172.21.15.32:0/87314737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c101090 0x7fac4c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.181+0000 7fac52144640 1 -- 172.21.15.32:0/87314737 >> 172.21.15.32:0/87314737 conn(0x7fac4c078000 msgr2=0x7fac4c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 -- 172.21.15.32:0/87314737 shutdown_connections 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 -- 172.21.15.32:0/87314737 wait complete. 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 Processor -- start 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 -- start start 2024-04-23T18:15:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.182+0000 7fac52144640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c106160 0x7fac4c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 0x7fac4c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c111080 con 0x7fac4c101090 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c110f00 con 0x7fac4c19d010 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c111200 con 0x7fac4c106160 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40954/0 (socket says 172.21.15.32:40954) 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 -- 172.21.15.32:0/3161201038 learned_addr learned my addr 172.21.15.32:0/3161201038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4affd640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c106160 0x7fac4c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4bfff640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 0x7fac4c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 -- 172.21.15.32:0/3161201038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c106160 msgr2=0x7fac4c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c106160 0x7fac4c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 -- 172.21.15.32:0/3161201038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 msgr2=0x7fac4c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 0x7fac4c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 -- 172.21.15.32:0/3161201038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c19db00 con 0x7fac4c101090 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4bfff640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 0x7fac4c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac4b7fe640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00e9f0 tx=0x7fac3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00cd80 con 0x7fac4c101090 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c19ddf0 con 0x7fac4c101090 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c0108e0 con 0x7fac4c101090 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c018980 con 0x7fac4c101090 2024-04-23T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.183+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c1a55b0 con 0x7fac4c101090 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.184+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4c068e00 con 0x7fac4c101090 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.185+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fac3c010430 con 0x7fac4c101090 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.186+0000 7fac48ff9640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 0x7fac340a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.186+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fac3c0d0ce0 con 0x7fac4c101090 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.186+0000 7fac4affd640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 0x7fac340a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.187+0000 7fac4affd640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 0x7fac340a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.191+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac3c09d260 con 0x7fac4c101090 2024-04-23T18:15:12.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.378+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fac4c101aa0 con 0x7fac4c101090 2024-04-23T18:15:12.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.378+0000 7fac48ff9640 1 -- 172.21.15.32:0/3161201038 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fac3c09cb60 con 0x7fac4c101090 2024-04-23T18:15:12.372 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:15:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 msgr2=0x7fac340a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 0x7fac340a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40031400 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 msgr2=0x7fac4c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00e9f0 tx=0x7fac3c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 shutdown_connections 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c19d010 0x7fac4c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac4c106160 0x7fac4c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fac340a3db0 0x7fac340a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 --2- 172.21.15.32:0/3161201038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac4c101090 0x7fac4c198750 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 >> 172.21.15.32:0/3161201038 conn(0x7fac4c078000 msgr2=0x7fac4c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 shutdown_connections 2024-04-23T18:15:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.380+0000 7fac52144640 1 -- 172.21.15.32:0/3161201038 wait complete. 2024-04-23T18:15:12.386 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-23T18:15:12.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.493+0000 7ff434ce1640 1 -- 172.21.15.32:0/3495601421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff430106ed0 msgr2=0x7ff4301072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.493+0000 7ff434ce1640 1 --2- 172.21.15.32:0/3495601421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff430106ed0 0x7ff4301072b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff420009900 tx=0x7ff42002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 -- 172.21.15.32:0/3495601421 shutdown_connections 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 --2- 172.21.15.32:0/3495601421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff43010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 --2- 172.21.15.32:0/3495601421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4301010a0 0x7ff430101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 --2- 172.21.15.32:0/3495601421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff430106ed0 0x7ff4301072b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 -- 172.21.15.32:0/3495601421 >> 172.21.15.32:0/3495601421 conn(0x7ff430077c40 msgr2=0x7ff430078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 -- 172.21.15.32:0/3495601421 shutdown_connections 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 -- 172.21.15.32:0/3495601421 wait complete. 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 Processor -- start 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.494+0000 7ff434ce1640 1 -- start start 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff430198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff430106ed0 0x7ff43019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff430112220 con 0x7ff4301010a0 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4301120a0 con 0x7ff430106ed0 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff434ce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4301123a0 con 0x7ff430101a40 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40972/0 (socket says 172.21.15.32:40972) 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 -- 172.21.15.32:0/643780131 learned_addr learned my addr 172.21.15.32:0/643780131 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42ed9d640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff430106ed0 0x7ff43019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42dd9b640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff430198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 -- 172.21.15.32:0/643780131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 msgr2=0x7ff430198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff430198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 -- 172.21.15.32:0/643780131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff430106ed0 msgr2=0x7ff43019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff430106ed0 0x7ff43019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 -- 172.21.15.32:0/643780131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43019d650 con 0x7ff4301010a0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42dd9b640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff430198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff42e59c640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff420005940 tx=0x7ff4200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.495+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42003f030 con 0x7ff4301010a0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.496+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff420003cb0 con 0x7ff4301010a0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.496+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43019d8e0 con 0x7ff4301010a0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.496+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420035070 con 0x7ff4301010a0 2024-04-23T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.496+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4301a51d0 con 0x7ff4301010a0 2024-04-23T18:15:12.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.497+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff430068e00 con 0x7ff4301010a0 2024-04-23T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.498+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7ff420042030 con 0x7ff4301010a0 2024-04-23T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.498+0000 7ff41f7fe640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 0x7ff4180a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.498+0000 7ff42dd9b640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 0x7ff4180a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.499+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff4200f11d0 con 0x7ff4301010a0 2024-04-23T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.499+0000 7ff42dd9b640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 0x7ff4180a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff4300781d0 tx=0x7ff424005d90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.503+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff4200bd750 con 0x7ff4301010a0 2024-04-23T18:15:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3640191705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4153327933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.673 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1180793208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:12 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3161201038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.687+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff430109360 con 0x7ff4180a3db0 2024-04-23T18:15:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.695+0000 7ff41f7fe640 1 -- 172.21.15.32:0/643780131 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257964 (secure 0 0 0) 0x7ff430109360 con 0x7ff4180a3db0 2024-04-23T18:15:12.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:12.694 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:15:12.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 msgr2=0x7ff4180a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 0x7ff4180a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff4300781d0 tx=0x7ff424005d90 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 msgr2=0x7ff430198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff420005940 tx=0x7ff4200042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 shutdown_connections 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff430106ed0 0x7ff43019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff430101a40 0x7ff430198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4180a3db0 0x7ff4180a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 --2- 172.21.15.32:0/643780131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4301010a0 0x7ff430198660 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 >> 172.21.15.32:0/643780131 conn(0x7ff430077c40 msgr2=0x7ff4300ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 shutdown_connections 2024-04-23T18:15:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.698+0000 7ff434ce1640 1 -- 172.21.15.32:0/643780131 wait complete. 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3640191705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.702 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4153327933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:12.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:12.704 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.704 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1180793208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.704 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:12 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3161201038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.704 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:15:11.379790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184776,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":976,"num_read_kb":646,"num_write":472,"num_write_kb":4375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181900,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377724,"kb_used_data":11912,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564100,"statfs":{"total":1151940427776,"available":1151553638400,"internally_reserved":0,"allocated":12197888,"data_stored":8488756,"data_compressed":175306,"data_compressed_allocated":225280,"data_compressed_original":2727936,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984406"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428784+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:15:07.428784+0000","last_peered":"2024-04-23T18:15:07.428784+0000","last_clean":"2024-04-23T18:15:07.428784+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:15:07.428784+0000","last_undegraded":"2024-04-23T18:15:07.428784+0000","last_fullsized":"2024-04-23T18:15:07.428784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131689+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:15:08.131689+0000","last_peered":"2024-04-23T18:15:08.131689+0000","last_clean":"2024-04-23T18:15:08.131689+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:15:08.131689+0000","last_undegraded":"2024-04-23T18:15:08.131689+0000","last_fullsized":"2024-04-23T18:15:08.131689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131604+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:15:08.131604+0000","last_peered":"2024-04-23T18:15:08.131604+0000","last_clean":"2024-04-23T18:15:08.131604+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:15:08.131604+0000","last_undegraded":"2024-04-23T18:15:08.131604+0000","last_fullsized":"2024-04-23T18:15:08.131604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232198+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:15:08.232198+0000","last_peered":"2024-04-23T18:15:08.232198+0000","last_clean":"2024-04-23T18:15:08.232198+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:15:08.232198+0000","last_undegraded":"2024-04-23T18:15:08.232198+0000","last_fullsized":"2024-04-23T18:15:08.232198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460058+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:15:07.460058+0000","last_peered":"2024-04-23T18:15:07.460058+0000","last_clean":"2024-04-23T18:15:07.460058+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:15:07.460058+0000","last_undegraded":"2024-04-23T18:15:07.460058+0000","last_fullsized":"2024-04-23T18:15:07.460058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470680+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:15:07.470680+0000","last_peered":"2024-04-23T18:15:07.470680+0000","last_clean":"2024-04-23T18:15:07.470680+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:15:07.470680+0000","last_undegraded":"2024-04-23T18:15:07.470680+0000","last_fullsized":"2024-04-23T18:15:07.470680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367559+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:15:07.367559+0000","last_peered":"2024-04-23T18:15:07.367559+0000","last_clean":"2024-04-23T18:15:07.367559+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:15:07.367559+0000","last_undegraded":"2024-04-23T18:15:07.367559+0000","last_fullsized":"2024-04-23T18:15:07.367559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459808+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:15:07.459808+0000","last_peered":"2024-04-23T18:15:07.459808+0000","last_clean":"2024-04-23T18:15:07.459808+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:15:07.459808+0000","last_undegraded":"2024-04-23T18:15:07.459808+0000","last_fullsized":"2024-04-23T18:15:07.459808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619887+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:15:07.619887+0000","last_peered":"2024-04-23T18:15:07.619887+0000","last_clean":"2024-04-23T18:15:07.619887+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:15:07.619887+0000","last_undegraded":"2024-04-23T18:15:07.619887+0000","last_fullsized":"2024-04-23T18:15:07.619887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470972+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:15:07.470972+0000","last_peered":"2024-04-23T18:15:07.470972+0000","last_clean":"2024-04-23T18:15:07.470972+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:15:07.470972+0000","last_undegraded":"2024-04-23T18:15:07.470972+0000","last_fullsized":"2024-04-23T18:15:07.470972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368053+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:15:07.368053+0000","last_peered":"2024-04-23T18:15:07.368053+0000","last_clean":"2024-04-23T18:15:07.368053+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:15:07.368053+0000","last_undegraded":"2024-04-23T18:15:07.368053+0000","last_fullsized":"2024-04-23T18:15:07.368053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232094+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:15:08.232094+0000","last_peered":"2024-04-23T18:15:08.232094+0000","last_clean":"2024-04-23T18:15:08.232094+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:15:08.232094+0000","last_undegraded":"2024-04-23T18:15:08.232094+0000","last_fullsized":"2024-04-23T18:15:08.232094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743908+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:15:07.743908+0000","last_peered":"2024-04-23T18:15:07.743908+0000","last_clean":"2024-04-23T18:15:07.743908+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:15:07.743908+0000","last_undegraded":"2024-04-23T18:15:07.743908+0000","last_fullsized":"2024-04-23T18:15:07.743908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131054+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:15:08.131054+0000","last_peered":"2024-04-23T18:15:08.131054+0000","last_clean":"2024-04-23T18:15:08.131054+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:15:08.131054+0000","last_undegraded":"2024-04-23T18:15:08.131054+0000","last_fullsized":"2024-04-23T18:15:08.131054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131043+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:15:08.131043+0000","last_peered":"2024-04-23T18:15:08.131043+0000","last_clean":"2024-04-23T18:15:08.131043+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:15:08.131043+0000","last_undegraded":"2024-04-23T18:15:08.131043+0000","last_fullsized":"2024-04-23T18:15:08.131043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619089+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:15:07.619089+0000","last_peered":"2024-04-23T18:15:07.619089+0000","last_clean":"2024-04-23T18:15:07.619089+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:15:07.619089+0000","last_undegraded":"2024-04-23T18:15:07.619089+0000","last_fullsized":"2024-04-23T18:15:07.619089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428915+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:15:07.428915+0000","last_peered":"2024-04-23T18:15:07.428915+0000","last_clean":"2024-04-23T18:15:07.428915+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:15:07.428915+0000","last_undegraded":"2024-04-23T18:15:07.428915+0000","last_fullsized":"2024-04-23T18:15:07.428915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366363+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:15:07.366363+0000","last_peered":"2024-04-23T18:15:07.366363+0000","last_clean":"2024-04-23T18:15:07.366363+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:15:07.366363+0000","last_undegraded":"2024-04-23T18:15:07.366363+0000","last_fullsized":"2024-04-23T18:15:07.366363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371397+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:15:07.371397+0000","last_peered":"2024-04-23T18:15:07.371397+0000","last_clean":"2024-04-23T18:15:07.371397+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:15:07.371397+0000","last_undegraded":"2024-04-23T18:15:07.371397+0000","last_fullsized":"2024-04-23T18:15:07.371397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366290+0000","last_change":"2024-04-23T18:13:57.299210+0000","last_active":"2024-04-23T18:15:07.366290+0000","last_peered":"2024-04-23T18:15:07.366290+0000","last_clean":"2024-04-23T18:15:07.366290+0000","last_became_active":"2024-04-23T18:13:57.292381+0000","last_became_peered":"2024-04-23T18:13:57.292381+0000","last_unstale":"2024-04-23T18:15:07.366290+0000","last_undegraded":"2024-04-23T18:15:07.366290+0000","last_fullsized":"2024-04-23T18:15:07.366290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:42.677179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460388+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:15:07.460388+0000","last_peered":"2024-04-23T18:15:07.460388+0000","last_clean":"2024-04-23T18:15:07.460388+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:15:07.460388+0000","last_undegraded":"2024-04-23T18:15:07.460388+0000","last_fullsized":"2024-04-23T18:15:07.460388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460664+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:15:07.460664+0000","last_peered":"2024-04-23T18:15:07.460664+0000","last_clean":"2024-04-23T18:15:07.460664+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:15:07.460664+0000","last_undegraded":"2024-04-23T18:15:07.460664+0000","last_fullsized":"2024-04-23T18:15:07.460664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131959+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:15:08.131959+0000","last_peered":"2024-04-23T18:15:08.131959+0000","last_clean":"2024-04-23T18:15:08.131959+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:15:08.131959+0000","last_undegraded":"2024-04-23T18:15:08.131959+0000","last_fullsized":"2024-04-23T18:15:08.131959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428758+0000","last_change":"2024-04-23T18:13:57.204788+0000","last_active":"2024-04-23T18:15:07.428758+0000","last_peered":"2024-04-23T18:15:07.428758+0000","last_clean":"2024-04-23T18:15:07.428758+0000","last_became_active":"2024-04-23T18:13:57.199938+0000","last_became_peered":"2024-04-23T18:13:57.199938+0000","last_unstale":"2024-04-23T18:15:07.428758+0000","last_undegraded":"2024-04-23T18:15:07.428758+0000","last_fullsized":"2024-04-23T18:15:07.428758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:10.411355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619400+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:15:07.619400+0000","last_peered":"2024-04-23T18:15:07.619400+0000","last_clean":"2024-04-23T18:15:07.619400+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:15:07.619400+0000","last_undegraded":"2024-04-23T18:15:07.619400+0000","last_fullsized":"2024-04-23T18:15:07.619400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460411+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:15:07.460411+0000","last_peered":"2024-04-23T18:15:07.460411+0000","last_clean":"2024-04-23T18:15:07.460411+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:15:07.460411+0000","last_undegraded":"2024-04-23T18:15:07.460411+0000","last_fullsized":"2024-04-23T18:15:07.460411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468083+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:15:07.468083+0000","last_peered":"2024-04-23T18:15:07.468083+0000","last_clean":"2024-04-23T18:15:07.468083+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:15:07.468083+0000","last_undegraded":"2024-04-23T18:15:07.468083+0000","last_fullsized":"2024-04-23T18:15:07.468083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'9","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231444+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:15:08.231444+0000","last_peered":"2024-04-23T18:15:08.231444+0000","last_clean":"2024-04-23T18:15:08.231444+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:15:08.231444+0000","last_undegraded":"2024-04-23T18:15:08.231444+0000","last_fullsized":"2024-04-23T18:15:08.231444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460016+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:15:07.460016+0000","last_peered":"2024-04-23T18:15:07.460016+0000","last_clean":"2024-04-23T18:15:07.460016+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:15:07.460016+0000","last_undegraded":"2024-04-23T18:15:07.460016+0000","last_fullsized":"2024-04-23T18:15:07.460016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470608+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:15:07.470608+0000","last_peered":"2024-04-23T18:15:07.470608+0000","last_clean":"2024-04-23T18:15:07.470608+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:15:07.470608+0000","last_undegraded":"2024-04-23T18:15:07.470608+0000","last_fullsized":"2024-04-23T18:15:07.470608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467899+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:15:07.467899+0000","last_peered":"2024-04-23T18:15:07.467899+0000","last_clean":"2024-04-23T18:15:07.467899+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:15:07.467899+0000","last_undegraded":"2024-04-23T18:15:07.467899+0000","last_fullsized":"2024-04-23T18:15:07.467899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743874+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:15:07.743874+0000","last_peered":"2024-04-23T18:15:07.743874+0000","last_clean":"2024-04-23T18:15:07.743874+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:15:07.743874+0000","last_undegraded":"2024-04-23T18:15:07.743874+0000","last_fullsized":"2024-04-23T18:15:07.743874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459741+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:15:07.459741+0000","last_peered":"2024-04-23T18:15:07.459741+0000","last_clean":"2024-04-23T18:15:07.459741+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:15:07.459741+0000","last_undegraded":"2024-04-23T18:15:07.459741+0000","last_fullsized":"2024-04-23T18:15:07.459741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467700+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:15:07.467700+0000","last_peered":"2024-04-23T18:15:07.467700+0000","last_clean":"2024-04-23T18:15:07.467700+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:15:07.467700+0000","last_undegraded":"2024-04-23T18:15:07.467700+0000","last_fullsized":"2024-04-23T18:15:07.467700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366070+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:15:07.366070+0000","last_peered":"2024-04-23T18:15:07.366070+0000","last_clean":"2024-04-23T18:15:07.366070+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:15:07.366070+0000","last_undegraded":"2024-04-23T18:15:07.366070+0000","last_fullsized":"2024-04-23T18:15:07.366070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470986+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:15:07.470986+0000","last_peered":"2024-04-23T18:15:07.470986+0000","last_clean":"2024-04-23T18:15:07.470986+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:15:07.470986+0000","last_undegraded":"2024-04-23T18:15:07.470986+0000","last_fullsized":"2024-04-23T18:15:07.470986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619758+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:15:07.619758+0000","last_peered":"2024-04-23T18:15:07.619758+0000","last_clean":"2024-04-23T18:15:07.619758+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:15:07.619758+0000","last_undegraded":"2024-04-23T18:15:07.619758+0000","last_fullsized":"2024-04-23T18:15:07.619758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367926+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:15:07.367926+0000","last_peered":"2024-04-23T18:15:07.367926+0000","last_clean":"2024-04-23T18:15:07.367926+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:15:07.367926+0000","last_undegraded":"2024-04-23T18:15:07.367926+0000","last_fullsized":"2024-04-23T18:15:07.367926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131914+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:15:08.131914+0000","last_peered":"2024-04-23T18:15:08.131914+0000","last_clean":"2024-04-23T18:15:08.131914+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:15:08.131914+0000","last_undegraded":"2024-04-23T18:15:08.131914+0000","last_fullsized":"2024-04-23T18:15:08.131914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468168+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:15:07.468168+0000","last_peered":"2024-04-23T18:15:07.468168+0000","last_clean":"2024-04-23T18:15:07.468168+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:15:07.468168+0000","last_undegraded":"2024-04-23T18:15:07.468168+0000","last_fullsized":"2024-04-23T18:15:07.468168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619569+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.619569+0000","last_peered":"2024-04-23T18:15:07.619569+0000","last_clean":"2024-04-23T18:15:07.619569+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:15:07.619569+0000","last_undegraded":"2024-04-23T18:15:07.619569+0000","last_fullsized":"2024-04-23T18:15:07.619569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231237+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:15:08.231237+0000","last_peered":"2024-04-23T18:15:08.231237+0000","last_clean":"2024-04-23T18:15:08.231237+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:15:08.231237+0000","last_undegraded":"2024-04-23T18:15:08.231237+0000","last_fullsized":"2024-04-23T18:15:08.231237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63985,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":187,"num_write":36,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460234+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:15:07.460234+0000","last_peered":"2024-04-23T18:15:07.460234+0000","last_clean":"2024-04-23T18:15:07.460234+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:15:07.460234+0000","last_undegraded":"2024-04-23T18:15:07.460234+0000","last_fullsized":"2024-04-23T18:15:07.460234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371185+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:15:07.371185+0000","last_peered":"2024-04-23T18:15:07.371185+0000","last_clean":"2024-04-23T18:15:07.371185+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:15:07.371185+0000","last_undegraded":"2024-04-23T18:15:07.371185+0000","last_fullsized":"2024-04-23T18:15:07.371185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743914+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:15:07.743914+0000","last_peered":"2024-04-23T18:15:07.743914+0000","last_clean":"2024-04-23T18:15:07.743914+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:15:07.743914+0000","last_undegraded":"2024-04-23T18:15:07.743914+0000","last_fullsized":"2024-04-23T18:15:07.743914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467867+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:15:07.467867+0000","last_peered":"2024-04-23T18:15:07.467867+0000","last_clean":"2024-04-23T18:15:07.467867+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:15:07.467867+0000","last_undegraded":"2024-04-23T18:15:07.467867+0000","last_fullsized":"2024-04-23T18:15:07.467867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368083+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.368083+0000","last_peered":"2024-04-23T18:15:07.368083+0000","last_clean":"2024-04-23T18:15:07.368083+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:15:07.368083+0000","last_undegraded":"2024-04-23T18:15:07.368083+0000","last_fullsized":"2024-04-23T18:15:07.368083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":149,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232046+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:15:08.232046+0000","last_peered":"2024-04-23T18:15:08.232046+0000","last_clean":"2024-04-23T18:15:08.232046+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:15:08.232046+0000","last_undegraded":"2024-04-23T18:15:08.232046+0000","last_fullsized":"2024-04-23T18:15:08.232046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367523+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:15:07.367523+0000","last_peered":"2024-04-23T18:15:07.367523+0000","last_clean":"2024-04-23T18:15:07.367523+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:15:07.367523+0000","last_undegraded":"2024-04-23T18:15:07.367523+0000","last_fullsized":"2024-04-23T18:15:07.367523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459987+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:15:07.459987+0000","last_peered":"2024-04-23T18:15:07.459987+0000","last_clean":"2024-04-23T18:15:07.459987+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:15:07.459987+0000","last_undegraded":"2024-04-23T18:15:07.459987+0000","last_fullsized":"2024-04-23T18:15:07.459987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467708+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:15:07.467708+0000","last_peered":"2024-04-23T18:15:07.467708+0000","last_clean":"2024-04-23T18:15:07.467708+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:15:07.467708+0000","last_undegraded":"2024-04-23T18:15:07.467708+0000","last_fullsized":"2024-04-23T18:15:07.467708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460312+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:15:07.460312+0000","last_peered":"2024-04-23T18:15:07.460312+0000","last_clean":"2024-04-23T18:15:07.460312+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:15:07.460312+0000","last_undegraded":"2024-04-23T18:15:07.460312+0000","last_fullsized":"2024-04-23T18:15:07.460312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428679+0000","last_change":"2024-04-23T18:13:57.292094+0000","last_active":"2024-04-23T18:15:07.428679+0000","last_peered":"2024-04-23T18:15:07.428679+0000","last_clean":"2024-04-23T18:15:07.428679+0000","last_became_active":"2024-04-23T18:13:57.291761+0000","last_became_peered":"2024-04-23T18:13:57.291761+0000","last_unstale":"2024-04-23T18:15:07.428679+0000","last_undegraded":"2024-04-23T18:15:07.428679+0000","last_fullsized":"2024-04-23T18:15:07.428679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:38:31.544598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366373+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:15:07.366373+0000","last_peered":"2024-04-23T18:15:07.366373+0000","last_clean":"2024-04-23T18:15:07.366373+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:15:07.366373+0000","last_undegraded":"2024-04-23T18:15:07.366373+0000","last_fullsized":"2024-04-23T18:15:07.366373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470956+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:15:07.470956+0000","last_peered":"2024-04-23T18:15:07.470956+0000","last_clean":"2024-04-23T18:15:07.470956+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:15:07.470956+0000","last_undegraded":"2024-04-23T18:15:07.470956+0000","last_fullsized":"2024-04-23T18:15:07.470956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368585+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:15:07.368585+0000","last_peered":"2024-04-23T18:15:07.368585+0000","last_clean":"2024-04-23T18:15:07.368585+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:15:07.368585+0000","last_undegraded":"2024-04-23T18:15:07.368585+0000","last_fullsized":"2024-04-23T18:15:07.368585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470804+0000","last_change":"2024-04-23T18:13:57.202859+0000","last_active":"2024-04-23T18:15:07.470804+0000","last_peered":"2024-04-23T18:15:07.470804+0000","last_clean":"2024-04-23T18:15:07.470804+0000","last_became_active":"2024-04-23T18:13:57.202395+0000","last_became_peered":"2024-04-23T18:13:57.202395+0000","last_unstale":"2024-04-23T18:15:07.470804+0000","last_undegraded":"2024-04-23T18:15:07.470804+0000","last_fullsized":"2024-04-23T18:15:07.470804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:17.871010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467943+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:15:07.467943+0000","last_peered":"2024-04-23T18:15:07.467943+0000","last_clean":"2024-04-23T18:15:07.467943+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:15:07.467943+0000","last_undegraded":"2024-04-23T18:15:07.467943+0000","last_fullsized":"2024-04-23T18:15:07.467943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743768+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:15:07.743768+0000","last_peered":"2024-04-23T18:15:07.743768+0000","last_clean":"2024-04-23T18:15:07.743768+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:15:07.743768+0000","last_undegraded":"2024-04-23T18:15:07.743768+0000","last_fullsized":"2024-04-23T18:15:07.743768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460210+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:15:07.460210+0000","last_peered":"2024-04-23T18:15:07.460210+0000","last_clean":"2024-04-23T18:15:07.460210+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:15:07.460210+0000","last_undegraded":"2024-04-23T18:15:07.460210+0000","last_fullsized":"2024-04-23T18:15:07.460210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232092+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:15:08.232092+0000","last_peered":"2024-04-23T18:15:08.232092+0000","last_clean":"2024-04-23T18:15:08.232092+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:15:08.232092+0000","last_undegraded":"2024-04-23T18:15:08.232092+0000","last_fullsized":"2024-04-23T18:15:08.232092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367609+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:15:07.367609+0000","last_peered":"2024-04-23T18:15:07.367609+0000","last_clean":"2024-04-23T18:15:07.367609+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:15:07.367609+0000","last_undegraded":"2024-04-23T18:15:07.367609+0000","last_fullsized":"2024-04-23T18:15:07.367609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743700+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:15:07.743700+0000","last_peered":"2024-04-23T18:15:07.743700+0000","last_clean":"2024-04-23T18:15:07.743700+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:15:07.743700+0000","last_undegraded":"2024-04-23T18:15:07.743700+0000","last_fullsized":"2024-04-23T18:15:07.743700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367785+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:15:07.367785+0000","last_peered":"2024-04-23T18:15:07.367785+0000","last_clean":"2024-04-23T18:15:07.367785+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:15:07.367785+0000","last_undegraded":"2024-04-23T18:15:07.367785+0000","last_fullsized":"2024-04-23T18:15:07.367785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460339+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:15:07.460339+0000","last_peered":"2024-04-23T18:15:07.460339+0000","last_clean":"2024-04-23T18:15:07.460339+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:15:07.460339+0000","last_undegraded":"2024-04-23T18:15:07.460339+0000","last_fullsized":"2024-04-23T18:15:07.460339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744119+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:15:07.744119+0000","last_peered":"2024-04-23T18:15:07.744119+0000","last_clean":"2024-04-23T18:15:07.744119+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:15:07.744119+0000","last_undegraded":"2024-04-23T18:15:07.744119+0000","last_fullsized":"2024-04-23T18:15:07.744119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'45","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.582411+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:15:07.582411+0000","last_peered":"2024-04-23T18:15:07.582411+0000","last_clean":"2024-04-23T18:15:07.582411+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:15:07.582411+0000","last_undegraded":"2024-04-23T18:15:07.582411+0000","last_fullsized":"2024-04-23T18:15:07.582411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460514+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:15:07.460514+0000","last_peered":"2024-04-23T18:15:07.460514+0000","last_clean":"2024-04-23T18:15:07.460514+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:15:07.460514+0000","last_undegraded":"2024-04-23T18:15:07.460514+0000","last_fullsized":"2024-04-23T18:15:07.460514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232315+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:15:08.232315+0000","last_peered":"2024-04-23T18:15:08.232315+0000","last_clean":"2024-04-23T18:15:08.232315+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:15:08.232315+0000","last_undegraded":"2024-04-23T18:15:08.232315+0000","last_fullsized":"2024-04-23T18:15:08.232315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131760+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:15:08.131760+0000","last_peered":"2024-04-23T18:15:08.131760+0000","last_clean":"2024-04-23T18:15:08.131760+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:15:08.131760+0000","last_undegraded":"2024-04-23T18:15:08.131760+0000","last_fullsized":"2024-04-23T18:15:08.131760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619167+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:15:07.619167+0000","last_peered":"2024-04-23T18:15:07.619167+0000","last_clean":"2024-04-23T18:15:07.619167+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:15:07.619167+0000","last_undegraded":"2024-04-23T18:15:07.619167+0000","last_fullsized":"2024-04-23T18:15:07.619167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459984+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:15:07.459984+0000","last_peered":"2024-04-23T18:15:07.459984+0000","last_clean":"2024-04-23T18:15:07.459984+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:15:07.459984+0000","last_undegraded":"2024-04-23T18:15:07.459984+0000","last_fullsized":"2024-04-23T18:15:07.459984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459981+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:15:07.459981+0000","last_peered":"2024-04-23T18:15:07.459981+0000","last_clean":"2024-04-23T18:15:07.459981+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:15:07.459981+0000","last_undegraded":"2024-04-23T18:15:07.459981+0000","last_fullsized":"2024-04-23T18:15:07.459981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428476+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:15:07.428476+0000","last_peered":"2024-04-23T18:15:07.428476+0000","last_clean":"2024-04-23T18:15:07.428476+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:15:07.428476+0000","last_undegraded":"2024-04-23T18:15:07.428476+0000","last_fullsized":"2024-04-23T18:15:07.428476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428685+0000","last_change":"2024-04-23T18:13:57.207521+0000","last_active":"2024-04-23T18:15:07.428685+0000","last_peered":"2024-04-23T18:15:07.428685+0000","last_clean":"2024-04-23T18:15:07.428685+0000","last_became_active":"2024-04-23T18:13:57.201507+0000","last_became_peered":"2024-04-23T18:13:57.201507+0000","last_unstale":"2024-04-23T18:15:07.428685+0000","last_undegraded":"2024-04-23T18:15:07.428685+0000","last_fullsized":"2024-04-23T18:15:07.428685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:35.254723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460384+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:15:07.460384+0000","last_peered":"2024-04-23T18:15:07.460384+0000","last_clean":"2024-04-23T18:15:07.460384+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:15:07.460384+0000","last_undegraded":"2024-04-23T18:15:07.460384+0000","last_fullsized":"2024-04-23T18:15:07.460384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460148+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:15:07.460148+0000","last_peered":"2024-04-23T18:15:07.460148+0000","last_clean":"2024-04-23T18:15:07.460148+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:15:07.460148+0000","last_undegraded":"2024-04-23T18:15:07.460148+0000","last_fullsized":"2024-04-23T18:15:07.460148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367860+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:15:07.367860+0000","last_peered":"2024-04-23T18:15:07.367860+0000","last_clean":"2024-04-23T18:15:07.367860+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:15:07.367860+0000","last_undegraded":"2024-04-23T18:15:07.367860+0000","last_fullsized":"2024-04-23T18:15:07.367860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619248+0000","last_change":"2024-04-23T18:13:57.206276+0000","last_active":"2024-04-23T18:15:07.619248+0000","last_peered":"2024-04-23T18:15:07.619248+0000","last_clean":"2024-04-23T18:15:07.619248+0000","last_became_active":"2024-04-23T18:13:57.202137+0000","last_became_peered":"2024-04-23T18:13:57.202137+0000","last_unstale":"2024-04-23T18:15:07.619248+0000","last_undegraded":"2024-04-23T18:15:07.619248+0000","last_fullsized":"2024-04-23T18:15:07.619248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:12.998331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367944+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:15:07.367944+0000","last_peered":"2024-04-23T18:15:07.367944+0000","last_clean":"2024-04-23T18:15:07.367944+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:15:07.367944+0000","last_undegraded":"2024-04-23T18:15:07.367944+0000","last_fullsized":"2024-04-23T18:15:07.367944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.577849+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:15:07.577849+0000","last_peered":"2024-04-23T18:15:07.577849+0000","last_clean":"2024-04-23T18:15:07.577849+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:15:07.577849+0000","last_undegraded":"2024-04-23T18:15:07.577849+0000","last_fullsized":"2024-04-23T18:15:07.577849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367840+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:15:07.367840+0000","last_peered":"2024-04-23T18:15:07.367840+0000","last_clean":"2024-04-23T18:15:07.367840+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:15:07.367840+0000","last_undegraded":"2024-04-23T18:15:07.367840+0000","last_fullsized":"2024-04-23T18:15:07.367840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744110+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:15:07.744110+0000","last_peered":"2024-04-23T18:15:07.744110+0000","last_clean":"2024-04-23T18:15:07.744110+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:15:07.744110+0000","last_undegraded":"2024-04-23T18:15:07.744110+0000","last_fullsized":"2024-04-23T18:15:07.744110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367824+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:15:07.367824+0000","last_peered":"2024-04-23T18:15:07.367824+0000","last_clean":"2024-04-23T18:15:07.367824+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:15:07.367824+0000","last_undegraded":"2024-04-23T18:15:07.367824+0000","last_fullsized":"2024-04-23T18:15:07.367824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428772+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:15:07.428772+0000","last_peered":"2024-04-23T18:15:07.428772+0000","last_clean":"2024-04-23T18:15:07.428772+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:15:07.428772+0000","last_undegraded":"2024-04-23T18:15:07.428772+0000","last_fullsized":"2024-04-23T18:15:07.428772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371128+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:15:07.371128+0000","last_peered":"2024-04-23T18:15:07.371128+0000","last_clean":"2024-04-23T18:15:07.371128+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:15:07.371128+0000","last_undegraded":"2024-04-23T18:15:07.371128+0000","last_fullsized":"2024-04-23T18:15:07.371128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371055+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:15:07.371055+0000","last_peered":"2024-04-23T18:15:07.371055+0000","last_clean":"2024-04-23T18:15:07.371055+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:15:07.371055+0000","last_undegraded":"2024-04-23T18:15:07.371055+0000","last_fullsized":"2024-04-23T18:15:07.371055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131355+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:15:08.131355+0000","last_peered":"2024-04-23T18:15:08.131355+0000","last_clean":"2024-04-23T18:15:08.131355+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:15:08.131355+0000","last_undegraded":"2024-04-23T18:15:08.131355+0000","last_fullsized":"2024-04-23T18:15:08.131355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743963+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:15:07.743963+0000","last_peered":"2024-04-23T18:15:07.743963+0000","last_clean":"2024-04-23T18:15:07.743963+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:15:07.743963+0000","last_undegraded":"2024-04-23T18:15:07.743963+0000","last_fullsized":"2024-04-23T18:15:07.743963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428702+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:15:07.428702+0000","last_peered":"2024-04-23T18:15:07.428702+0000","last_clean":"2024-04-23T18:15:07.428702+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:15:07.428702+0000","last_undegraded":"2024-04-23T18:15:07.428702+0000","last_fullsized":"2024-04-23T18:15:07.428702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371506+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:15:07.371506+0000","last_peered":"2024-04-23T18:15:07.371506+0000","last_clean":"2024-04-23T18:15:07.371506+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:15:07.371506+0000","last_undegraded":"2024-04-23T18:15:07.371506+0000","last_fullsized":"2024-04-23T18:15:07.371506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460394+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:15:07.460394+0000","last_peered":"2024-04-23T18:15:07.460394+0000","last_clean":"2024-04-23T18:15:07.460394+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:15:07.460394+0000","last_undegraded":"2024-04-23T18:15:07.460394+0000","last_fullsized":"2024-04-23T18:15:07.460394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467697+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:15:07.467697+0000","last_peered":"2024-04-23T18:15:07.467697+0000","last_clean":"2024-04-23T18:15:07.467697+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:15:07.467697+0000","last_undegraded":"2024-04-23T18:15:07.467697+0000","last_fullsized":"2024-04-23T18:15:07.467697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459881+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:15:07.459881+0000","last_peered":"2024-04-23T18:15:07.459881+0000","last_clean":"2024-04-23T18:15:07.459881+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:15:07.459881+0000","last_undegraded":"2024-04-23T18:15:07.459881+0000","last_fullsized":"2024-04-23T18:15:07.459881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619351+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:15:07.619351+0000","last_peered":"2024-04-23T18:15:07.619351+0000","last_clean":"2024-04-23T18:15:07.619351+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:15:07.619351+0000","last_undegraded":"2024-04-23T18:15:07.619351+0000","last_fullsized":"2024-04-23T18:15:07.619351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743931+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:15:07.743931+0000","last_peered":"2024-04-23T18:15:07.743931+0000","last_clean":"2024-04-23T18:15:07.743931+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:15:07.743931+0000","last_undegraded":"2024-04-23T18:15:07.743931+0000","last_fullsized":"2024-04-23T18:15:07.743931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428525+0000","last_change":"2024-04-23T18:13:57.199427+0000","last_active":"2024-04-23T18:15:07.428525+0000","last_peered":"2024-04-23T18:15:07.428525+0000","last_clean":"2024-04-23T18:15:07.428525+0000","last_became_active":"2024-04-23T18:13:57.199001+0000","last_became_peered":"2024-04-23T18:13:57.199001+0000","last_unstale":"2024-04-23T18:15:07.428525+0000","last_undegraded":"2024-04-23T18:15:07.428525+0000","last_fullsized":"2024-04-23T18:15:07.428525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:14.788961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594174,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":840,"num_read_kb":527,"num_write":289,"num_write_kb":1435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1164,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1782522,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":469494,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1257092,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":600140,"data_compressed":9316,"data_compressed_allocated":12288,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":1062561,"data_compressed":38944,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":533479,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":461492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":533723,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":994246,"data_compressed":28188,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1193704,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":197695,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:15:12.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:15:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 -- 172.21.15.32:0/260517809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c81013e0 msgr2=0x7f14c8106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 --2- 172.21.15.32:0/260517809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c81013e0 0x7f14c8106bb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009900 tx=0x7f14b002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 -- 172.21.15.32:0/260517809 shutdown_connections 2024-04-23T18:15:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 --2- 172.21.15.32:0/260517809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8107290 0x7f14c810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 --2- 172.21.15.32:0/260517809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c81013e0 0x7f14c8106bb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 --2- 172.21.15.32:0/260517809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c8100a30 0x7f14c8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 -- 172.21.15.32:0/260517809 >> 172.21.15.32:0/260517809 conn(0x7f14c8077300 msgr2=0x7f14c8077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 -- 172.21.15.32:0/260517809 shutdown_connections 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 -- 172.21.15.32:0/260517809 wait complete. 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.853+0000 7f14ccce1640 1 Processor -- start 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 -- start start 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 0x7f14c81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 0x7f14c819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14c81121b0 con 0x7f14c8107290 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8112030 con 0x7f14c8100a30 2024-04-23T18:15:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14ccce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8112330 con 0x7f14c819cfb0 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40990/0 (socket says 172.21.15.32:40990) 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 -- 172.21.15.32:0/1016526838 learned_addr learned my addr 172.21.15.32:0/1016526838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14c659c640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 0x7f14c81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14c6d9d640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 0x7f14c819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 -- 172.21.15.32:0/1016526838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 msgr2=0x7f14c819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 0x7f14c819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 -- 172.21.15.32:0/1016526838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 msgr2=0x7f14c81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 0x7f14c81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14bdd9b640 1 -- 172.21.15.32:0/1016526838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c819daa0 con 0x7f14c8107290 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14c659c640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 0x7f14c81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.854+0000 7f14c6d9d640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 0x7f14c819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14bdd9b640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006010 tx=0x7f14b0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b003f030 con 0x7f14c8107290 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c819dd30 con 0x7f14c8107290 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b0003aa0 con 0x7f14c8107290 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0035070 con 0x7f14c8107290 2024-04-23T18:15:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.855+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c81a54f0 con 0x7f14c8107290 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.856+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c8069250 con 0x7f14c8107290 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.857+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f14b0042030 con 0x7f14c8107290 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.857+0000 7f14bffff640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 0x7f14b40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.858+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f14b00f0e80 con 0x7f14c8107290 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.858+0000 7f14c659c640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 0x7f14b40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.858+0000 7f14c659c640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 0x7f14b40a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b80058c0 tx=0x7f14b8005850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:12.862+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14b00bd400 con 0x7f14c8107290 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:12.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3640191705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: Updating smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4153327933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:12.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:12.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:12.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:15:12.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1180793208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:12.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:12 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3161201038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:13.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.048+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14c81013e0 con 0x7f14b40a3db0 2024-04-23T18:15:13.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.055+0000 7f14bffff640 1 -- 172.21.15.32:0/1016526838 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257964 (secure 0 0 0) 0x7f14c81013e0 con 0x7f14b40a3db0 2024-04-23T18:15:13.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.058+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 msgr2=0x7f14b40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.058+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 0x7f14b40a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b80058c0 tx=0x7f14b8005850 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.058+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 msgr2=0x7f14c8198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.058+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006010 tx=0x7f14b0004060 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.058+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 shutdown_connections 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14c819cfb0 0x7f14c819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f14b40a3db0 0x7f14b40a6270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c8107290 0x7f14c8198c30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 --2- 172.21.15.32:0/1016526838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14c8100a30 0x7f14c81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 >> 172.21.15.32:0/1016526838 conn(0x7f14c8077300 msgr2=0x7f14c80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 shutdown_connections 2024-04-23T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.059+0000 7f14ccce1640 1 -- 172.21.15.32:0/1016526838 wait complete. 2024-04-23T18:15:13.064 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:15:11.379790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184776,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":976,"num_read_kb":646,"num_write":472,"num_write_kb":4375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181900,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377724,"kb_used_data":11912,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564100,"statfs":{"total":1151940427776,"available":1151553638400,"internally_reserved":0,"allocated":12197888,"data_stored":8488756,"data_compressed":175306,"data_compressed_allocated":225280,"data_compressed_original":2727936,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984406"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428784+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:15:07.428784+0000","last_peered":"2024-04-23T18:15:07.428784+0000","last_clean":"2024-04-23T18:15:07.428784+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:15:07.428784+0000","last_undegraded":"2024-04-23T18:15:07.428784+0000","last_fullsized":"2024-04-23T18:15:07.428784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131689+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:15:08.131689+0000","last_peered":"2024-04-23T18:15:08.131689+0000","last_clean":"2024-04-23T18:15:08.131689+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:15:08.131689+0000","last_undegraded":"2024-04-23T18:15:08.131689+0000","last_fullsized":"2024-04-23T18:15:08.131689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131604+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:15:08.131604+0000","last_peered":"2024-04-23T18:15:08.131604+0000","last_clean":"2024-04-23T18:15:08.131604+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:15:08.131604+0000","last_undegraded":"2024-04-23T18:15:08.131604+0000","last_fullsized":"2024-04-23T18:15:08.131604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232198+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:15:08.232198+0000","last_peered":"2024-04-23T18:15:08.232198+0000","last_clean":"2024-04-23T18:15:08.232198+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:15:08.232198+0000","last_undegraded":"2024-04-23T18:15:08.232198+0000","last_fullsized":"2024-04-23T18:15:08.232198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460058+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:15:07.460058+0000","last_peered":"2024-04-23T18:15:07.460058+0000","last_clean":"2024-04-23T18:15:07.460058+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:15:07.460058+0000","last_undegraded":"2024-04-23T18:15:07.460058+0000","last_fullsized":"2024-04-23T18:15:07.460058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470680+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:15:07.470680+0000","last_peered":"2024-04-23T18:15:07.470680+0000","last_clean":"2024-04-23T18:15:07.470680+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:15:07.470680+0000","last_undegraded":"2024-04-23T18:15:07.470680+0000","last_fullsized":"2024-04-23T18:15:07.470680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367559+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:15:07.367559+0000","last_peered":"2024-04-23T18:15:07.367559+0000","last_clean":"2024-04-23T18:15:07.367559+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:15:07.367559+0000","last_undegraded":"2024-04-23T18:15:07.367559+0000","last_fullsized":"2024-04-23T18:15:07.367559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459808+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:15:07.459808+0000","last_peered":"2024-04-23T18:15:07.459808+0000","last_clean":"2024-04-23T18:15:07.459808+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:15:07.459808+0000","last_undegraded":"2024-04-23T18:15:07.459808+0000","last_fullsized":"2024-04-23T18:15:07.459808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619887+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:15:07.619887+0000","last_peered":"2024-04-23T18:15:07.619887+0000","last_clean":"2024-04-23T18:15:07.619887+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:15:07.619887+0000","last_undegraded":"2024-04-23T18:15:07.619887+0000","last_fullsized":"2024-04-23T18:15:07.619887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470972+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:15:07.470972+0000","last_peered":"2024-04-23T18:15:07.470972+0000","last_clean":"2024-04-23T18:15:07.470972+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:15:07.470972+0000","last_undegraded":"2024-04-23T18:15:07.470972+0000","last_fullsized":"2024-04-23T18:15:07.470972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368053+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:15:07.368053+0000","last_peered":"2024-04-23T18:15:07.368053+0000","last_clean":"2024-04-23T18:15:07.368053+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:15:07.368053+0000","last_undegraded":"2024-04-23T18:15:07.368053+0000","last_fullsized":"2024-04-23T18:15:07.368053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232094+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:15:08.232094+0000","last_peered":"2024-04-23T18:15:08.232094+0000","last_clean":"2024-04-23T18:15:08.232094+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:15:08.232094+0000","last_undegraded":"2024-04-23T18:15:08.232094+0000","last_fullsized":"2024-04-23T18:15:08.232094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743908+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:15:07.743908+0000","last_peered":"2024-04-23T18:15:07.743908+0000","last_clean":"2024-04-23T18:15:07.743908+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:15:07.743908+0000","last_undegraded":"2024-04-23T18:15:07.743908+0000","last_fullsized":"2024-04-23T18:15:07.743908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131054+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:15:08.131054+0000","last_peered":"2024-04-23T18:15:08.131054+0000","last_clean":"2024-04-23T18:15:08.131054+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:15:08.131054+0000","last_undegraded":"2024-04-23T18:15:08.131054+0000","last_fullsized":"2024-04-23T18:15:08.131054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131043+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:15:08.131043+0000","last_peered":"2024-04-23T18:15:08.131043+0000","last_clean":"2024-04-23T18:15:08.131043+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:15:08.131043+0000","last_undegraded":"2024-04-23T18:15:08.131043+0000","last_fullsized":"2024-04-23T18:15:08.131043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619089+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:15:07.619089+0000","last_peered":"2024-04-23T18:15:07.619089+0000","last_clean":"2024-04-23T18:15:07.619089+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:15:07.619089+0000","last_undegraded":"2024-04-23T18:15:07.619089+0000","last_fullsized":"2024-04-23T18:15:07.619089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428915+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:15:07.428915+0000","last_peered":"2024-04-23T18:15:07.428915+0000","last_clean":"2024-04-23T18:15:07.428915+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:15:07.428915+0000","last_undegraded":"2024-04-23T18:15:07.428915+0000","last_fullsized":"2024-04-23T18:15:07.428915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366363+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:15:07.366363+0000","last_peered":"2024-04-23T18:15:07.366363+0000","last_clean":"2024-04-23T18:15:07.366363+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:15:07.366363+0000","last_undegraded":"2024-04-23T18:15:07.366363+0000","last_fullsized":"2024-04-23T18:15:07.366363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371397+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:15:07.371397+0000","last_peered":"2024-04-23T18:15:07.371397+0000","last_clean":"2024-04-23T18:15:07.371397+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:15:07.371397+0000","last_undegraded":"2024-04-23T18:15:07.371397+0000","last_fullsized":"2024-04-23T18:15:07.371397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366290+0000","last_change":"2024-04-23T18:13:57.299210+0000","last_active":"2024-04-23T18:15:07.366290+0000","last_peered":"2024-04-23T18:15:07.366290+0000","last_clean":"2024-04-23T18:15:07.366290+0000","last_became_active":"2024-04-23T18:13:57.292381+0000","last_became_peered":"2024-04-23T18:13:57.292381+0000","last_unstale":"2024-04-23T18:15:07.366290+0000","last_undegraded":"2024-04-23T18:15:07.366290+0000","last_fullsized":"2024-04-23T18:15:07.366290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:42.677179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460388+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:15:07.460388+0000","last_peered":"2024-04-23T18:15:07.460388+0000","last_clean":"2024-04-23T18:15:07.460388+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:15:07.460388+0000","last_undegraded":"2024-04-23T18:15:07.460388+0000","last_fullsized":"2024-04-23T18:15:07.460388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460664+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:15:07.460664+0000","last_peered":"2024-04-23T18:15:07.460664+0000","last_clean":"2024-04-23T18:15:07.460664+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:15:07.460664+0000","last_undegraded":"2024-04-23T18:15:07.460664+0000","last_fullsized":"2024-04-23T18:15:07.460664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131959+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:15:08.131959+0000","last_peered":"2024-04-23T18:15:08.131959+0000","last_clean":"2024-04-23T18:15:08.131959+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:15:08.131959+0000","last_undegraded":"2024-04-23T18:15:08.131959+0000","last_fullsized":"2024-04-23T18:15:08.131959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428758+0000","last_change":"2024-04-23T18:13:57.204788+0000","last_active":"2024-04-23T18:15:07.428758+0000","last_peered":"2024-04-23T18:15:07.428758+0000","last_clean":"2024-04-23T18:15:07.428758+0000","last_became_active":"2024-04-23T18:13:57.199938+0000","last_became_peered":"2024-04-23T18:13:57.199938+0000","last_unstale":"2024-04-23T18:15:07.428758+0000","last_undegraded":"2024-04-23T18:15:07.428758+0000","last_fullsized":"2024-04-23T18:15:07.428758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:10.411355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619400+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:15:07.619400+0000","last_peered":"2024-04-23T18:15:07.619400+0000","last_clean":"2024-04-23T18:15:07.619400+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:15:07.619400+0000","last_undegraded":"2024-04-23T18:15:07.619400+0000","last_fullsized":"2024-04-23T18:15:07.619400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460411+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:15:07.460411+0000","last_peered":"2024-04-23T18:15:07.460411+0000","last_clean":"2024-04-23T18:15:07.460411+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:15:07.460411+0000","last_undegraded":"2024-04-23T18:15:07.460411+0000","last_fullsized":"2024-04-23T18:15:07.460411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468083+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:15:07.468083+0000","last_peered":"2024-04-23T18:15:07.468083+0000","last_clean":"2024-04-23T18:15:07.468083+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:15:07.468083+0000","last_undegraded":"2024-04-23T18:15:07.468083+0000","last_fullsized":"2024-04-23T18:15:07.468083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'9","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231444+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:15:08.231444+0000","last_peered":"2024-04-23T18:15:08.231444+0000","last_clean":"2024-04-23T18:15:08.231444+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:15:08.231444+0000","last_undegraded":"2024-04-23T18:15:08.231444+0000","last_fullsized":"2024-04-23T18:15:08.231444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460016+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:15:07.460016+0000","last_peered":"2024-04-23T18:15:07.460016+0000","last_clean":"2024-04-23T18:15:07.460016+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:15:07.460016+0000","last_undegraded":"2024-04-23T18:15:07.460016+0000","last_fullsized":"2024-04-23T18:15:07.460016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470608+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:15:07.470608+0000","last_peered":"2024-04-23T18:15:07.470608+0000","last_clean":"2024-04-23T18:15:07.470608+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:15:07.470608+0000","last_undegraded":"2024-04-23T18:15:07.470608+0000","last_fullsized":"2024-04-23T18:15:07.470608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467899+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:15:07.467899+0000","last_peered":"2024-04-23T18:15:07.467899+0000","last_clean":"2024-04-23T18:15:07.467899+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:15:07.467899+0000","last_undegraded":"2024-04-23T18:15:07.467899+0000","last_fullsized":"2024-04-23T18:15:07.467899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743874+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:15:07.743874+0000","last_peered":"2024-04-23T18:15:07.743874+0000","last_clean":"2024-04-23T18:15:07.743874+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:15:07.743874+0000","last_undegraded":"2024-04-23T18:15:07.743874+0000","last_fullsized":"2024-04-23T18:15:07.743874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459741+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:15:07.459741+0000","last_peered":"2024-04-23T18:15:07.459741+0000","last_clean":"2024-04-23T18:15:07.459741+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:15:07.459741+0000","last_undegraded":"2024-04-23T18:15:07.459741+0000","last_fullsized":"2024-04-23T18:15:07.459741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467700+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:15:07.467700+0000","last_peered":"2024-04-23T18:15:07.467700+0000","last_clean":"2024-04-23T18:15:07.467700+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:15:07.467700+0000","last_undegraded":"2024-04-23T18:15:07.467700+0000","last_fullsized":"2024-04-23T18:15:07.467700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366070+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:15:07.366070+0000","last_peered":"2024-04-23T18:15:07.366070+0000","last_clean":"2024-04-23T18:15:07.366070+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:15:07.366070+0000","last_undegraded":"2024-04-23T18:15:07.366070+0000","last_fullsized":"2024-04-23T18:15:07.366070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470986+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:15:07.470986+0000","last_peered":"2024-04-23T18:15:07.470986+0000","last_clean":"2024-04-23T18:15:07.470986+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:15:07.470986+0000","last_undegraded":"2024-04-23T18:15:07.470986+0000","last_fullsized":"2024-04-23T18:15:07.470986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619758+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:15:07.619758+0000","last_peered":"2024-04-23T18:15:07.619758+0000","last_clean":"2024-04-23T18:15:07.619758+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:15:07.619758+0000","last_undegraded":"2024-04-23T18:15:07.619758+0000","last_fullsized":"2024-04-23T18:15:07.619758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367926+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:15:07.367926+0000","last_peered":"2024-04-23T18:15:07.367926+0000","last_clean":"2024-04-23T18:15:07.367926+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:15:07.367926+0000","last_undegraded":"2024-04-23T18:15:07.367926+0000","last_fullsized":"2024-04-23T18:15:07.367926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131914+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:15:08.131914+0000","last_peered":"2024-04-23T18:15:08.131914+0000","last_clean":"2024-04-23T18:15:08.131914+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:15:08.131914+0000","last_undegraded":"2024-04-23T18:15:08.131914+0000","last_fullsized":"2024-04-23T18:15:08.131914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468168+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:15:07.468168+0000","last_peered":"2024-04-23T18:15:07.468168+0000","last_clean":"2024-04-23T18:15:07.468168+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:15:07.468168+0000","last_undegraded":"2024-04-23T18:15:07.468168+0000","last_fullsized":"2024-04-23T18:15:07.468168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619569+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.619569+0000","last_peered":"2024-04-23T18:15:07.619569+0000","last_clean":"2024-04-23T18:15:07.619569+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:15:07.619569+0000","last_undegraded":"2024-04-23T18:15:07.619569+0000","last_fullsized":"2024-04-23T18:15:07.619569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231237+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:15:08.231237+0000","last_peered":"2024-04-23T18:15:08.231237+0000","last_clean":"2024-04-23T18:15:08.231237+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:15:08.231237+0000","last_undegraded":"2024-04-23T18:15:08.231237+0000","last_fullsized":"2024-04-23T18:15:08.231237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63985,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":187,"num_write":36,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460234+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:15:07.460234+0000","last_peered":"2024-04-23T18:15:07.460234+0000","last_clean":"2024-04-23T18:15:07.460234+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:15:07.460234+0000","last_undegraded":"2024-04-23T18:15:07.460234+0000","last_fullsized":"2024-04-23T18:15:07.460234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371185+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:15:07.371185+0000","last_peered":"2024-04-23T18:15:07.371185+0000","last_clean":"2024-04-23T18:15:07.371185+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:15:07.371185+0000","last_undegraded":"2024-04-23T18:15:07.371185+0000","last_fullsized":"2024-04-23T18:15:07.371185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743914+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:15:07.743914+0000","last_peered":"2024-04-23T18:15:07.743914+0000","last_clean":"2024-04-23T18:15:07.743914+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:15:07.743914+0000","last_undegraded":"2024-04-23T18:15:07.743914+0000","last_fullsized":"2024-04-23T18:15:07.743914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467867+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:15:07.467867+0000","last_peered":"2024-04-23T18:15:07.467867+0000","last_clean":"2024-04-23T18:15:07.467867+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:15:07.467867+0000","last_undegraded":"2024-04-23T18:15:07.467867+0000","last_fullsized":"2024-04-23T18:15:07.467867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368083+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.368083+0000","last_peered":"2024-04-23T18:15:07.368083+0000","last_clean":"2024-04-23T18:15:07.368083+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:15:07.368083+0000","last_undegraded":"2024-04-23T18:15:07.368083+0000","last_fullsized":"2024-04-23T18:15:07.368083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":149,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232046+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:15:08.232046+0000","last_peered":"2024-04-23T18:15:08.232046+0000","last_clean":"2024-04-23T18:15:08.232046+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:15:08.232046+0000","last_undegraded":"2024-04-23T18:15:08.232046+0000","last_fullsized":"2024-04-23T18:15:08.232046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367523+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:15:07.367523+0000","last_peered":"2024-04-23T18:15:07.367523+0000","last_clean":"2024-04-23T18:15:07.367523+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:15:07.367523+0000","last_undegraded":"2024-04-23T18:15:07.367523+0000","last_fullsized":"2024-04-23T18:15:07.367523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459987+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:15:07.459987+0000","last_peered":"2024-04-23T18:15:07.459987+0000","last_clean":"2024-04-23T18:15:07.459987+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:15:07.459987+0000","last_undegraded":"2024-04-23T18:15:07.459987+0000","last_fullsized":"2024-04-23T18:15:07.459987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467708+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:15:07.467708+0000","last_peered":"2024-04-23T18:15:07.467708+0000","last_clean":"2024-04-23T18:15:07.467708+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:15:07.467708+0000","last_undegraded":"2024-04-23T18:15:07.467708+0000","last_fullsized":"2024-04-23T18:15:07.467708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460312+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:15:07.460312+0000","last_peered":"2024-04-23T18:15:07.460312+0000","last_clean":"2024-04-23T18:15:07.460312+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:15:07.460312+0000","last_undegraded":"2024-04-23T18:15:07.460312+0000","last_fullsized":"2024-04-23T18:15:07.460312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428679+0000","last_change":"2024-04-23T18:13:57.292094+0000","last_active":"2024-04-23T18:15:07.428679+0000","last_peered":"2024-04-23T18:15:07.428679+0000","last_clean":"2024-04-23T18:15:07.428679+0000","last_became_active":"2024-04-23T18:13:57.291761+0000","last_became_peered":"2024-04-23T18:13:57.291761+0000","last_unstale":"2024-04-23T18:15:07.428679+0000","last_undegraded":"2024-04-23T18:15:07.428679+0000","last_fullsized":"2024-04-23T18:15:07.428679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:38:31.544598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366373+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:15:07.366373+0000","last_peered":"2024-04-23T18:15:07.366373+0000","last_clean":"2024-04-23T18:15:07.366373+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:15:07.366373+0000","last_undegraded":"2024-04-23T18:15:07.366373+0000","last_fullsized":"2024-04-23T18:15:07.366373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470956+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:15:07.470956+0000","last_peered":"2024-04-23T18:15:07.470956+0000","last_clean":"2024-04-23T18:15:07.470956+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:15:07.470956+0000","last_undegraded":"2024-04-23T18:15:07.470956+0000","last_fullsized":"2024-04-23T18:15:07.470956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368585+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:15:07.368585+0000","last_peered":"2024-04-23T18:15:07.368585+0000","last_clean":"2024-04-23T18:15:07.368585+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:15:07.368585+0000","last_undegraded":"2024-04-23T18:15:07.368585+0000","last_fullsized":"2024-04-23T18:15:07.368585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470804+0000","last_change":"2024-04-23T18:13:57.202859+0000","last_active":"2024-04-23T18:15:07.470804+0000","last_peered":"2024-04-23T18:15:07.470804+0000","last_clean":"2024-04-23T18:15:07.470804+0000","last_became_active":"2024-04-23T18:13:57.202395+0000","last_became_peered":"2024-04-23T18:13:57.202395+0000","last_unstale":"2024-04-23T18:15:07.470804+0000","last_undegraded":"2024-04-23T18:15:07.470804+0000","last_fullsized":"2024-04-23T18:15:07.470804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:17.871010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467943+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:15:07.467943+0000","last_peered":"2024-04-23T18:15:07.467943+0000","last_clean":"2024-04-23T18:15:07.467943+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:15:07.467943+0000","last_undegraded":"2024-04-23T18:15:07.467943+0000","last_fullsized":"2024-04-23T18:15:07.467943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743768+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:15:07.743768+0000","last_peered":"2024-04-23T18:15:07.743768+0000","last_clean":"2024-04-23T18:15:07.743768+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:15:07.743768+0000","last_undegraded":"2024-04-23T18:15:07.743768+0000","last_fullsized":"2024-04-23T18:15:07.743768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460210+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:15:07.460210+0000","last_peered":"2024-04-23T18:15:07.460210+0000","last_clean":"2024-04-23T18:15:07.460210+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:15:07.460210+0000","last_undegraded":"2024-04-23T18:15:07.460210+0000","last_fullsized":"2024-04-23T18:15:07.460210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232092+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:15:08.232092+0000","last_peered":"2024-04-23T18:15:08.232092+0000","last_clean":"2024-04-23T18:15:08.232092+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:15:08.232092+0000","last_undegraded":"2024-04-23T18:15:08.232092+0000","last_fullsized":"2024-04-23T18:15:08.232092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367609+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:15:07.367609+0000","last_peered":"2024-04-23T18:15:07.367609+0000","last_clean":"2024-04-23T18:15:07.367609+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:15:07.367609+0000","last_undegraded":"2024-04-23T18:15:07.367609+0000","last_fullsized":"2024-04-23T18:15:07.367609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743700+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:15:07.743700+0000","last_peered":"2024-04-23T18:15:07.743700+0000","last_clean":"2024-04-23T18:15:07.743700+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:15:07.743700+0000","last_undegraded":"2024-04-23T18:15:07.743700+0000","last_fullsized":"2024-04-23T18:15:07.743700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367785+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:15:07.367785+0000","last_peered":"2024-04-23T18:15:07.367785+0000","last_clean":"2024-04-23T18:15:07.367785+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:15:07.367785+0000","last_undegraded":"2024-04-23T18:15:07.367785+0000","last_fullsized":"2024-04-23T18:15:07.367785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460339+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:15:07.460339+0000","last_peered":"2024-04-23T18:15:07.460339+0000","last_clean":"2024-04-23T18:15:07.460339+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:15:07.460339+0000","last_undegraded":"2024-04-23T18:15:07.460339+0000","last_fullsized":"2024-04-23T18:15:07.460339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744119+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:15:07.744119+0000","last_peered":"2024-04-23T18:15:07.744119+0000","last_clean":"2024-04-23T18:15:07.744119+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:15:07.744119+0000","last_undegraded":"2024-04-23T18:15:07.744119+0000","last_fullsized":"2024-04-23T18:15:07.744119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'45","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.582411+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:15:07.582411+0000","last_peered":"2024-04-23T18:15:07.582411+0000","last_clean":"2024-04-23T18:15:07.582411+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:15:07.582411+0000","last_undegraded":"2024-04-23T18:15:07.582411+0000","last_fullsized":"2024-04-23T18:15:07.582411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460514+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:15:07.460514+0000","last_peered":"2024-04-23T18:15:07.460514+0000","last_clean":"2024-04-23T18:15:07.460514+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:15:07.460514+0000","last_undegraded":"2024-04-23T18:15:07.460514+0000","last_fullsized":"2024-04-23T18:15:07.460514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232315+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:15:08.232315+0000","last_peered":"2024-04-23T18:15:08.232315+0000","last_clean":"2024-04-23T18:15:08.232315+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:15:08.232315+0000","last_undegraded":"2024-04-23T18:15:08.232315+0000","last_fullsized":"2024-04-23T18:15:08.232315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131760+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:15:08.131760+0000","last_peered":"2024-04-23T18:15:08.131760+0000","last_clean":"2024-04-23T18:15:08.131760+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:15:08.131760+0000","last_undegraded":"2024-04-23T18:15:08.131760+0000","last_fullsized":"2024-04-23T18:15:08.131760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619167+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:15:07.619167+0000","last_peered":"2024-04-23T18:15:07.619167+0000","last_clean":"2024-04-23T18:15:07.619167+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:15:07.619167+0000","last_undegraded":"2024-04-23T18:15:07.619167+0000","last_fullsized":"2024-04-23T18:15:07.619167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459984+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:15:07.459984+0000","last_peered":"2024-04-23T18:15:07.459984+0000","last_clean":"2024-04-23T18:15:07.459984+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:15:07.459984+0000","last_undegraded":"2024-04-23T18:15:07.459984+0000","last_fullsized":"2024-04-23T18:15:07.459984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459981+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:15:07.459981+0000","last_peered":"2024-04-23T18:15:07.459981+0000","last_clean":"2024-04-23T18:15:07.459981+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:15:07.459981+0000","last_undegraded":"2024-04-23T18:15:07.459981+0000","last_fullsized":"2024-04-23T18:15:07.459981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428476+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:15:07.428476+0000","last_peered":"2024-04-23T18:15:07.428476+0000","last_clean":"2024-04-23T18:15:07.428476+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:15:07.428476+0000","last_undegraded":"2024-04-23T18:15:07.428476+0000","last_fullsized":"2024-04-23T18:15:07.428476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428685+0000","last_change":"2024-04-23T18:13:57.207521+0000","last_active":"2024-04-23T18:15:07.428685+0000","last_peered":"2024-04-23T18:15:07.428685+0000","last_clean":"2024-04-23T18:15:07.428685+0000","last_became_active":"2024-04-23T18:13:57.201507+0000","last_became_peered":"2024-04-23T18:13:57.201507+0000","last_unstale":"2024-04-23T18:15:07.428685+0000","last_undegraded":"2024-04-23T18:15:07.428685+0000","last_fullsized":"2024-04-23T18:15:07.428685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:35.254723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460384+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:15:07.460384+0000","last_peered":"2024-04-23T18:15:07.460384+0000","last_clean":"2024-04-23T18:15:07.460384+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:15:07.460384+0000","last_undegraded":"2024-04-23T18:15:07.460384+0000","last_fullsized":"2024-04-23T18:15:07.460384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460148+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:15:07.460148+0000","last_peered":"2024-04-23T18:15:07.460148+0000","last_clean":"2024-04-23T18:15:07.460148+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:15:07.460148+0000","last_undegraded":"2024-04-23T18:15:07.460148+0000","last_fullsized":"2024-04-23T18:15:07.460148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367860+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:15:07.367860+0000","last_peered":"2024-04-23T18:15:07.367860+0000","last_clean":"2024-04-23T18:15:07.367860+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:15:07.367860+0000","last_undegraded":"2024-04-23T18:15:07.367860+0000","last_fullsized":"2024-04-23T18:15:07.367860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619248+0000","last_change":"2024-04-23T18:13:57.206276+0000","last_active":"2024-04-23T18:15:07.619248+0000","last_peered":"2024-04-23T18:15:07.619248+0000","last_clean":"2024-04-23T18:15:07.619248+0000","last_became_active":"2024-04-23T18:13:57.202137+0000","last_became_peered":"2024-04-23T18:13:57.202137+0000","last_unstale":"2024-04-23T18:15:07.619248+0000","last_undegraded":"2024-04-23T18:15:07.619248+0000","last_fullsized":"2024-04-23T18:15:07.619248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:12.998331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367944+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:15:07.367944+0000","last_peered":"2024-04-23T18:15:07.367944+0000","last_clean":"2024-04-23T18:15:07.367944+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:15:07.367944+0000","last_undegraded":"2024-04-23T18:15:07.367944+0000","last_fullsized":"2024-04-23T18:15:07.367944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.577849+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:15:07.577849+0000","last_peered":"2024-04-23T18:15:07.577849+0000","last_clean":"2024-04-23T18:15:07.577849+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:15:07.577849+0000","last_undegraded":"2024-04-23T18:15:07.577849+0000","last_fullsized":"2024-04-23T18:15:07.577849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367840+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:15:07.367840+0000","last_peered":"2024-04-23T18:15:07.367840+0000","last_clean":"2024-04-23T18:15:07.367840+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:15:07.367840+0000","last_undegraded":"2024-04-23T18:15:07.367840+0000","last_fullsized":"2024-04-23T18:15:07.367840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744110+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:15:07.744110+0000","last_peered":"2024-04-23T18:15:07.744110+0000","last_clean":"2024-04-23T18:15:07.744110+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:15:07.744110+0000","last_undegraded":"2024-04-23T18:15:07.744110+0000","last_fullsized":"2024-04-23T18:15:07.744110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367824+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:15:07.367824+0000","last_peered":"2024-04-23T18:15:07.367824+0000","last_clean":"2024-04-23T18:15:07.367824+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:15:07.367824+0000","last_undegraded":"2024-04-23T18:15:07.367824+0000","last_fullsized":"2024-04-23T18:15:07.367824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428772+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:15:07.428772+0000","last_peered":"2024-04-23T18:15:07.428772+0000","last_clean":"2024-04-23T18:15:07.428772+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:15:07.428772+0000","last_undegraded":"2024-04-23T18:15:07.428772+0000","last_fullsized":"2024-04-23T18:15:07.428772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371128+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:15:07.371128+0000","last_peered":"2024-04-23T18:15:07.371128+0000","last_clean":"2024-04-23T18:15:07.371128+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:15:07.371128+0000","last_undegraded":"2024-04-23T18:15:07.371128+0000","last_fullsized":"2024-04-23T18:15:07.371128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371055+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:15:07.371055+0000","last_peered":"2024-04-23T18:15:07.371055+0000","last_clean":"2024-04-23T18:15:07.371055+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:15:07.371055+0000","last_undegraded":"2024-04-23T18:15:07.371055+0000","last_fullsized":"2024-04-23T18:15:07.371055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131355+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:15:08.131355+0000","last_peered":"2024-04-23T18:15:08.131355+0000","last_clean":"2024-04-23T18:15:08.131355+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:15:08.131355+0000","last_undegraded":"2024-04-23T18:15:08.131355+0000","last_fullsized":"2024-04-23T18:15:08.131355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743963+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:15:07.743963+0000","last_peered":"2024-04-23T18:15:07.743963+0000","last_clean":"2024-04-23T18:15:07.743963+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:15:07.743963+0000","last_undegraded":"2024-04-23T18:15:07.743963+0000","last_fullsized":"2024-04-23T18:15:07.743963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428702+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:15:07.428702+0000","last_peered":"2024-04-23T18:15:07.428702+0000","last_clean":"2024-04-23T18:15:07.428702+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:15:07.428702+0000","last_undegraded":"2024-04-23T18:15:07.428702+0000","last_fullsized":"2024-04-23T18:15:07.428702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371506+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:15:07.371506+0000","last_peered":"2024-04-23T18:15:07.371506+0000","last_clean":"2024-04-23T18:15:07.371506+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:15:07.371506+0000","last_undegraded":"2024-04-23T18:15:07.371506+0000","last_fullsized":"2024-04-23T18:15:07.371506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460394+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:15:07.460394+0000","last_peered":"2024-04-23T18:15:07.460394+0000","last_clean":"2024-04-23T18:15:07.460394+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:15:07.460394+0000","last_undegraded":"2024-04-23T18:15:07.460394+0000","last_fullsized":"2024-04-23T18:15:07.460394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467697+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:15:07.467697+0000","last_peered":"2024-04-23T18:15:07.467697+0000","last_clean":"2024-04-23T18:15:07.467697+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:15:07.467697+0000","last_undegraded":"2024-04-23T18:15:07.467697+0000","last_fullsized":"2024-04-23T18:15:07.467697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459881+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:15:07.459881+0000","last_peered":"2024-04-23T18:15:07.459881+0000","last_clean":"2024-04-23T18:15:07.459881+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:15:07.459881+0000","last_undegraded":"2024-04-23T18:15:07.459881+0000","last_fullsized":"2024-04-23T18:15:07.459881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619351+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:15:07.619351+0000","last_peered":"2024-04-23T18:15:07.619351+0000","last_clean":"2024-04-23T18:15:07.619351+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:15:07.619351+0000","last_undegraded":"2024-04-23T18:15:07.619351+0000","last_fullsized":"2024-04-23T18:15:07.619351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743931+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:15:07.743931+0000","last_peered":"2024-04-23T18:15:07.743931+0000","last_clean":"2024-04-23T18:15:07.743931+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:15:07.743931+0000","last_undegraded":"2024-04-23T18:15:07.743931+0000","last_fullsized":"2024-04-23T18:15:07.743931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428525+0000","last_change":"2024-04-23T18:13:57.199427+0000","last_active":"2024-04-23T18:15:07.428525+0000","last_peered":"2024-04-23T18:15:07.428525+0000","last_clean":"2024-04-23T18:15:07.428525+0000","last_became_active":"2024-04-23T18:13:57.199001+0000","last_became_peered":"2024-04-23T18:13:57.199001+0000","last_unstale":"2024-04-23T18:15:07.428525+0000","last_undegraded":"2024-04-23T18:15:07.428525+0000","last_fullsized":"2024-04-23T18:15:07.428525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:14.788961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594174,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":840,"num_read_kb":527,"num_write":289,"num_write_kb":1435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1164,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1782522,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":469494,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1257092,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":600140,"data_compressed":9316,"data_compressed_allocated":12288,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":1062561,"data_compressed":38944,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":533479,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":461492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":533723,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":994246,"data_compressed":28188,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1193704,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":197695,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:15:13.068 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-23T18:15:13.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:15:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 -- 172.21.15.32:0/1934252223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c1031c0 msgr2=0x7fd26c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 --2- 172.21.15.32:0/1934252223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c1031c0 0x7fd26c1035a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c02a480 comp rx=0 tx=0).stop 2024-04-23T18:15:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 -- 172.21.15.32:0/1934252223 shutdown_connections 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 --2- 172.21.15.32:0/1934252223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c10c590 0x7fd26c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 --2- 172.21.15.32:0/1934252223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c103ae0 0x7fd26c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 --2- 172.21.15.32:0/1934252223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c1031c0 0x7fd26c1035a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 -- 172.21.15.32:0/1934252223 >> 172.21.15.32:0/1934252223 conn(0x7fd26c077f50 msgr2=0x7fd26c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 -- 172.21.15.32:0/1934252223 shutdown_connections 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.212+0000 7fd2718e1640 1 -- 172.21.15.32:0/1934252223 wait complete. 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 Processor -- start 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 -- start start 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c1031c0 0x7fd26c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c103ae0 0x7fd26c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd26c112230 con 0x7fd26c10c590 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd26c1120b0 con 0x7fd26c1031c0 2024-04-23T18:15:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd2718e1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd26c1123b0 con 0x7fd26c103ae0 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd26b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd263fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c103ae0 0x7fd26c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd26b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41018/0 (socket says 172.21.15.32:41018) 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd26b7fe640 1 -- 172.21.15.32:0/3458138105 learned_addr learned my addr 172.21.15.32:0/3458138105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.213+0000 7fd26affd640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c1031c0 0x7fd26c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 -- 172.21.15.32:0/3458138105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c103ae0 msgr2=0x7fd26c198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c103ae0 0x7fd26c198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 -- 172.21.15.32:0/3458138105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c1031c0 msgr2=0x7fd26c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c1031c0 0x7fd26c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 -- 172.21.15.32:0/3458138105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26c19d720 con 0x7fd26c10c590 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd26b7fe640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd26400cab0 tx=0x7fd26400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26400ae20 con 0x7fd26c10c590 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd26400f900 con 0x7fd26c10c590 2024-04-23T18:15:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26c19da10 con 0x7fd26c10c590 2024-04-23T18:15:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264018a50 con 0x7fd26c10c590 2024-04-23T18:15:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.214+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26c1a52a0 con 0x7fd26c10c590 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.215+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd26c069250 con 0x7fd26c10c590 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.216+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fd26400f450 con 0x7fd26c10c590 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.217+0000 7fd2708df640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 0x7fd2580a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.217+0000 7fd26affd640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 0x7fd2580a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.217+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fd2640d0b80 con 0x7fd26c10c590 2024-04-23T18:15:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.217+0000 7fd26affd640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 0x7fd2580a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd25c005ab0 tx=0x7fd25c005a20 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.221+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd264014030 con 0x7fd26c10c590 2024-04-23T18:15:13.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.405+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd26c062390 con 0x7fd2580a3e30 2024-04-23T18:15:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.413+0000 7fd2708df640 1 -- 172.21.15.32:0/3458138105 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257973 (secure 0 0 0) 0x7fd26c062390 con 0x7fd2580a3e30 2024-04-23T18:15:13.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:13.410 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.415+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 msgr2=0x7fd2580a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.415+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 0x7fd2580a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd25c005ab0 tx=0x7fd25c005a20 comp rx=0 tx=0).stop 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.415+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 msgr2=0x7fd26c19cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.415+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd26400cab0 tx=0x7fd26400cf70 comp rx=0 tx=0).stop 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 shutdown_connections 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2580a3e30 0x7fd2580a62f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd26c10c590 0x7fd26c19cff0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd26c103ae0 0x7fd26c198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 --2- 172.21.15.32:0/3458138105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd26c1031c0 0x7fd26c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 >> 172.21.15.32:0/3458138105 conn(0x7fd26c077f50 msgr2=0x7fd26c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 shutdown_connections 2024-04-23T18:15:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.416+0000 7fd2718e1640 1 -- 172.21.15.32:0/3458138105 wait complete. 2024-04-23T18:15:13.420 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-23T18:15:13.380203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184776,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":976,"num_read_kb":646,"num_write":472,"num_write_kb":4375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181900,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377740,"kb_used_data":11924,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564084,"statfs":{"total":1151940427776,"available":1151553622016,"internally_reserved":0,"allocated":12210176,"data_stored":8497654,"data_compressed":178192,"data_compressed_allocated":229376,"data_compressed_original":2736128,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.984819"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428784+0000","last_change":"2024-04-23T18:13:05.527960+0000","last_active":"2024-04-23T18:15:07.428784+0000","last_peered":"2024-04-23T18:15:07.428784+0000","last_clean":"2024-04-23T18:15:07.428784+0000","last_became_active":"2024-04-23T18:13:05.527611+0000","last_became_peered":"2024-04-23T18:13:05.527611+0000","last_unstale":"2024-04-23T18:15:07.428784+0000","last_undegraded":"2024-04-23T18:15:07.428784+0000","last_fullsized":"2024-04-23T18:15:07.428784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:07.089002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131739+0000","last_change":"2024-04-23T18:13:05.527855+0000","last_active":"2024-04-23T18:15:08.131739+0000","last_peered":"2024-04-23T18:15:08.131739+0000","last_clean":"2024-04-23T18:15:08.131739+0000","last_became_active":"2024-04-23T18:13:05.527395+0000","last_became_peered":"2024-04-23T18:13:05.527395+0000","last_unstale":"2024-04-23T18:15:08.131739+0000","last_undegraded":"2024-04-23T18:15:08.131739+0000","last_fullsized":"2024-04-23T18:15:08.131739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:54.403332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131653+0000","last_change":"2024-04-23T18:13:05.535500+0000","last_active":"2024-04-23T18:15:08.131653+0000","last_peered":"2024-04-23T18:15:08.131653+0000","last_clean":"2024-04-23T18:15:08.131653+0000","last_became_active":"2024-04-23T18:13:05.535118+0000","last_became_peered":"2024-04-23T18:13:05.535118+0000","last_unstale":"2024-04-23T18:15:08.131653+0000","last_undegraded":"2024-04-23T18:15:08.131653+0000","last_fullsized":"2024-04-23T18:15:08.131653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:18.878082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232198+0000","last_change":"2024-04-23T18:13:05.536678+0000","last_active":"2024-04-23T18:15:08.232198+0000","last_peered":"2024-04-23T18:15:08.232198+0000","last_clean":"2024-04-23T18:15:08.232198+0000","last_became_active":"2024-04-23T18:13:05.536316+0000","last_became_peered":"2024-04-23T18:13:05.536316+0000","last_unstale":"2024-04-23T18:15:08.232198+0000","last_undegraded":"2024-04-23T18:15:08.232198+0000","last_fullsized":"2024-04-23T18:15:08.232198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:35.002124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460058+0000","last_change":"2024-04-23T18:13:05.529655+0000","last_active":"2024-04-23T18:15:07.460058+0000","last_peered":"2024-04-23T18:15:07.460058+0000","last_clean":"2024-04-23T18:15:07.460058+0000","last_became_active":"2024-04-23T18:13:05.529291+0000","last_became_peered":"2024-04-23T18:13:05.529291+0000","last_unstale":"2024-04-23T18:15:07.460058+0000","last_undegraded":"2024-04-23T18:15:07.460058+0000","last_fullsized":"2024-04-23T18:15:07.460058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:27.442862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470680+0000","last_change":"2024-04-23T18:13:05.538049+0000","last_active":"2024-04-23T18:15:07.470680+0000","last_peered":"2024-04-23T18:15:07.470680+0000","last_clean":"2024-04-23T18:15:07.470680+0000","last_became_active":"2024-04-23T18:13:05.537717+0000","last_became_peered":"2024-04-23T18:13:05.537717+0000","last_unstale":"2024-04-23T18:15:07.470680+0000","last_undegraded":"2024-04-23T18:15:07.470680+0000","last_fullsized":"2024-04-23T18:15:07.470680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.581591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367559+0000","last_change":"2024-04-23T18:13:05.531936+0000","last_active":"2024-04-23T18:15:07.367559+0000","last_peered":"2024-04-23T18:15:07.367559+0000","last_clean":"2024-04-23T18:15:07.367559+0000","last_became_active":"2024-04-23T18:13:05.531588+0000","last_became_peered":"2024-04-23T18:13:05.531588+0000","last_unstale":"2024-04-23T18:15:07.367559+0000","last_undegraded":"2024-04-23T18:15:07.367559+0000","last_fullsized":"2024-04-23T18:15:07.367559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:45.765758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459808+0000","last_change":"2024-04-23T18:13:05.536628+0000","last_active":"2024-04-23T18:15:07.459808+0000","last_peered":"2024-04-23T18:15:07.459808+0000","last_clean":"2024-04-23T18:15:07.459808+0000","last_became_active":"2024-04-23T18:13:05.536317+0000","last_became_peered":"2024-04-23T18:13:05.536317+0000","last_unstale":"2024-04-23T18:15:07.459808+0000","last_undegraded":"2024-04-23T18:15:07.459808+0000","last_fullsized":"2024-04-23T18:15:07.459808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:30.328514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619887+0000","last_change":"2024-04-23T18:13:05.530323+0000","last_active":"2024-04-23T18:15:07.619887+0000","last_peered":"2024-04-23T18:15:07.619887+0000","last_clean":"2024-04-23T18:15:07.619887+0000","last_became_active":"2024-04-23T18:13:05.529954+0000","last_became_peered":"2024-04-23T18:13:05.529954+0000","last_unstale":"2024-04-23T18:15:07.619887+0000","last_undegraded":"2024-04-23T18:15:07.619887+0000","last_fullsized":"2024-04-23T18:15:07.619887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:19.128502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470972+0000","last_change":"2024-04-23T18:13:05.536103+0000","last_active":"2024-04-23T18:15:07.470972+0000","last_peered":"2024-04-23T18:15:07.470972+0000","last_clean":"2024-04-23T18:15:07.470972+0000","last_became_active":"2024-04-23T18:13:05.535736+0000","last_became_peered":"2024-04-23T18:13:05.535736+0000","last_unstale":"2024-04-23T18:15:07.470972+0000","last_undegraded":"2024-04-23T18:15:07.470972+0000","last_fullsized":"2024-04-23T18:15:07.470972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:47.365359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368053+0000","last_change":"2024-04-23T18:13:05.535559+0000","last_active":"2024-04-23T18:15:07.368053+0000","last_peered":"2024-04-23T18:15:07.368053+0000","last_clean":"2024-04-23T18:15:07.368053+0000","last_became_active":"2024-04-23T18:13:05.535224+0000","last_became_peered":"2024-04-23T18:13:05.535224+0000","last_unstale":"2024-04-23T18:15:07.368053+0000","last_undegraded":"2024-04-23T18:15:07.368053+0000","last_fullsized":"2024-04-23T18:15:07.368053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:02:34.362288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232094+0000","last_change":"2024-04-23T18:13:05.534218+0000","last_active":"2024-04-23T18:15:08.232094+0000","last_peered":"2024-04-23T18:15:08.232094+0000","last_clean":"2024-04-23T18:15:08.232094+0000","last_became_active":"2024-04-23T18:13:05.533754+0000","last_became_peered":"2024-04-23T18:13:05.533754+0000","last_unstale":"2024-04-23T18:15:08.232094+0000","last_undegraded":"2024-04-23T18:15:08.232094+0000","last_fullsized":"2024-04-23T18:15:08.232094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:51.518977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743908+0000","last_change":"2024-04-23T18:13:05.522313+0000","last_active":"2024-04-23T18:15:07.743908+0000","last_peered":"2024-04-23T18:15:07.743908+0000","last_clean":"2024-04-23T18:15:07.743908+0000","last_became_active":"2024-04-23T18:13:05.521701+0000","last_became_peered":"2024-04-23T18:13:05.521701+0000","last_unstale":"2024-04-23T18:15:07.743908+0000","last_undegraded":"2024-04-23T18:15:07.743908+0000","last_fullsized":"2024-04-23T18:15:07.743908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:38.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131054+0000","last_change":"2024-04-23T18:13:05.532684+0000","last_active":"2024-04-23T18:15:08.131054+0000","last_peered":"2024-04-23T18:15:08.131054+0000","last_clean":"2024-04-23T18:15:08.131054+0000","last_became_active":"2024-04-23T18:13:05.532376+0000","last_became_peered":"2024-04-23T18:13:05.532376+0000","last_unstale":"2024-04-23T18:15:08.131054+0000","last_undegraded":"2024-04-23T18:15:08.131054+0000","last_fullsized":"2024-04-23T18:15:08.131054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.162122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131043+0000","last_change":"2024-04-23T18:13:05.536538+0000","last_active":"2024-04-23T18:15:08.131043+0000","last_peered":"2024-04-23T18:15:08.131043+0000","last_clean":"2024-04-23T18:15:08.131043+0000","last_became_active":"2024-04-23T18:13:05.536176+0000","last_became_peered":"2024-04-23T18:13:05.536176+0000","last_unstale":"2024-04-23T18:15:08.131043+0000","last_undegraded":"2024-04-23T18:15:08.131043+0000","last_fullsized":"2024-04-23T18:15:08.131043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:25.549011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619089+0000","last_change":"2024-04-23T18:13:05.529050+0000","last_active":"2024-04-23T18:15:07.619089+0000","last_peered":"2024-04-23T18:15:07.619089+0000","last_clean":"2024-04-23T18:15:07.619089+0000","last_became_active":"2024-04-23T18:13:05.528310+0000","last_became_peered":"2024-04-23T18:13:05.528310+0000","last_unstale":"2024-04-23T18:15:07.619089+0000","last_undegraded":"2024-04-23T18:15:07.619089+0000","last_fullsized":"2024-04-23T18:15:07.619089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:22:31.840288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428915+0000","last_change":"2024-04-23T18:13:05.529462+0000","last_active":"2024-04-23T18:15:07.428915+0000","last_peered":"2024-04-23T18:15:07.428915+0000","last_clean":"2024-04-23T18:15:07.428915+0000","last_became_active":"2024-04-23T18:13:05.529159+0000","last_became_peered":"2024-04-23T18:13:05.529159+0000","last_unstale":"2024-04-23T18:15:07.428915+0000","last_undegraded":"2024-04-23T18:15:07.428915+0000","last_fullsized":"2024-04-23T18:15:07.428915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:59.630853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366363+0000","last_change":"2024-04-23T18:13:05.526462+0000","last_active":"2024-04-23T18:15:07.366363+0000","last_peered":"2024-04-23T18:15:07.366363+0000","last_clean":"2024-04-23T18:15:07.366363+0000","last_became_active":"2024-04-23T18:13:05.526164+0000","last_became_peered":"2024-04-23T18:13:05.526164+0000","last_unstale":"2024-04-23T18:15:07.366363+0000","last_undegraded":"2024-04-23T18:15:07.366363+0000","last_fullsized":"2024-04-23T18:15:07.366363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:33:08.162229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371397+0000","last_change":"2024-04-23T18:13:05.520506+0000","last_active":"2024-04-23T18:15:07.371397+0000","last_peered":"2024-04-23T18:15:07.371397+0000","last_clean":"2024-04-23T18:15:07.371397+0000","last_became_active":"2024-04-23T18:13:05.520130+0000","last_became_peered":"2024-04-23T18:13:05.520130+0000","last_unstale":"2024-04-23T18:15:07.371397+0000","last_undegraded":"2024-04-23T18:15:07.371397+0000","last_fullsized":"2024-04-23T18:15:07.371397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:13.252683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366290+0000","last_change":"2024-04-23T18:13:57.299210+0000","last_active":"2024-04-23T18:15:07.366290+0000","last_peered":"2024-04-23T18:15:07.366290+0000","last_clean":"2024-04-23T18:15:07.366290+0000","last_became_active":"2024-04-23T18:13:57.292381+0000","last_became_peered":"2024-04-23T18:13:57.292381+0000","last_unstale":"2024-04-23T18:15:07.366290+0000","last_undegraded":"2024-04-23T18:15:07.366290+0000","last_fullsized":"2024-04-23T18:15:07.366290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:42.677179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460388+0000","last_change":"2024-04-23T18:13:05.534721+0000","last_active":"2024-04-23T18:15:07.460388+0000","last_peered":"2024-04-23T18:15:07.460388+0000","last_clean":"2024-04-23T18:15:07.460388+0000","last_became_active":"2024-04-23T18:13:05.528586+0000","last_became_peered":"2024-04-23T18:13:05.528586+0000","last_unstale":"2024-04-23T18:15:07.460388+0000","last_undegraded":"2024-04-23T18:15:07.460388+0000","last_fullsized":"2024-04-23T18:15:07.460388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:17.602007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460664+0000","last_change":"2024-04-23T18:13:05.023838+0000","last_active":"2024-04-23T18:15:07.460664+0000","last_peered":"2024-04-23T18:15:07.460664+0000","last_clean":"2024-04-23T18:15:07.460664+0000","last_became_active":"2024-04-23T18:13:05.023436+0000","last_became_peered":"2024-04-23T18:13:05.023436+0000","last_unstale":"2024-04-23T18:15:07.460664+0000","last_undegraded":"2024-04-23T18:15:07.460664+0000","last_fullsized":"2024-04-23T18:15:07.460664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:10.445198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.132016+0000","last_change":"2024-04-23T18:13:05.530338+0000","last_active":"2024-04-23T18:15:08.132016+0000","last_peered":"2024-04-23T18:15:08.132016+0000","last_clean":"2024-04-23T18:15:08.132016+0000","last_became_active":"2024-04-23T18:13:05.529968+0000","last_became_peered":"2024-04-23T18:13:05.529968+0000","last_unstale":"2024-04-23T18:15:08.132016+0000","last_undegraded":"2024-04-23T18:15:08.132016+0000","last_fullsized":"2024-04-23T18:15:08.132016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:43.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428758+0000","last_change":"2024-04-23T18:13:57.204788+0000","last_active":"2024-04-23T18:15:07.428758+0000","last_peered":"2024-04-23T18:15:07.428758+0000","last_clean":"2024-04-23T18:15:07.428758+0000","last_became_active":"2024-04-23T18:13:57.199938+0000","last_became_peered":"2024-04-23T18:13:57.199938+0000","last_unstale":"2024-04-23T18:15:07.428758+0000","last_undegraded":"2024-04-23T18:15:07.428758+0000","last_fullsized":"2024-04-23T18:15:07.428758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:10.411355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619400+0000","last_change":"2024-04-23T18:13:05.530786+0000","last_active":"2024-04-23T18:15:07.619400+0000","last_peered":"2024-04-23T18:15:07.619400+0000","last_clean":"2024-04-23T18:15:07.619400+0000","last_became_active":"2024-04-23T18:13:05.530301+0000","last_became_peered":"2024-04-23T18:13:05.530301+0000","last_unstale":"2024-04-23T18:15:07.619400+0000","last_undegraded":"2024-04-23T18:15:07.619400+0000","last_fullsized":"2024-04-23T18:15:07.619400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:12.243592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460411+0000","last_change":"2024-04-23T18:13:05.024469+0000","last_active":"2024-04-23T18:15:07.460411+0000","last_peered":"2024-04-23T18:15:07.460411+0000","last_clean":"2024-04-23T18:15:07.460411+0000","last_became_active":"2024-04-23T18:13:05.024137+0000","last_became_peered":"2024-04-23T18:13:05.024137+0000","last_unstale":"2024-04-23T18:15:07.460411+0000","last_undegraded":"2024-04-23T18:15:07.460411+0000","last_fullsized":"2024-04-23T18:15:07.460411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:25.580028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468083+0000","last_change":"2024-04-23T18:13:05.531230+0000","last_active":"2024-04-23T18:15:07.468083+0000","last_peered":"2024-04-23T18:15:07.468083+0000","last_clean":"2024-04-23T18:15:07.468083+0000","last_became_active":"2024-04-23T18:13:05.530788+0000","last_became_peered":"2024-04-23T18:13:05.530788+0000","last_unstale":"2024-04-23T18:15:07.468083+0000","last_undegraded":"2024-04-23T18:15:07.468083+0000","last_fullsized":"2024-04-23T18:15:07.468083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:23.683749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'9","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231444+0000","last_change":"2024-04-23T18:13:05.024175+0000","last_active":"2024-04-23T18:15:08.231444+0000","last_peered":"2024-04-23T18:15:08.231444+0000","last_clean":"2024-04-23T18:15:08.231444+0000","last_became_active":"2024-04-23T18:13:05.023820+0000","last_became_peered":"2024-04-23T18:13:05.023820+0000","last_unstale":"2024-04-23T18:15:08.231444+0000","last_undegraded":"2024-04-23T18:15:08.231444+0000","last_fullsized":"2024-04-23T18:15:08.231444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:25.352409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460016+0000","last_change":"2024-04-23T18:13:05.528857+0000","last_active":"2024-04-23T18:15:07.460016+0000","last_peered":"2024-04-23T18:15:07.460016+0000","last_clean":"2024-04-23T18:15:07.460016+0000","last_became_active":"2024-04-23T18:13:05.528507+0000","last_became_peered":"2024-04-23T18:13:05.528507+0000","last_unstale":"2024-04-23T18:15:07.460016+0000","last_undegraded":"2024-04-23T18:15:07.460016+0000","last_fullsized":"2024-04-23T18:15:07.460016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:22.636125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470608+0000","last_change":"2024-04-23T18:13:05.020556+0000","last_active":"2024-04-23T18:15:07.470608+0000","last_peered":"2024-04-23T18:15:07.470608+0000","last_clean":"2024-04-23T18:15:07.470608+0000","last_became_active":"2024-04-23T18:13:05.020106+0000","last_became_peered":"2024-04-23T18:13:05.020106+0000","last_unstale":"2024-04-23T18:15:07.470608+0000","last_undegraded":"2024-04-23T18:15:07.470608+0000","last_fullsized":"2024-04-23T18:15:07.470608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:39.339584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467899+0000","last_change":"2024-04-23T18:13:05.536225+0000","last_active":"2024-04-23T18:15:07.467899+0000","last_peered":"2024-04-23T18:15:07.467899+0000","last_clean":"2024-04-23T18:15:07.467899+0000","last_became_active":"2024-04-23T18:13:05.535840+0000","last_became_peered":"2024-04-23T18:13:05.535840+0000","last_unstale":"2024-04-23T18:15:07.467899+0000","last_undegraded":"2024-04-23T18:15:07.467899+0000","last_fullsized":"2024-04-23T18:15:07.467899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:14.027838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743874+0000","last_change":"2024-04-23T18:13:05.025369+0000","last_active":"2024-04-23T18:15:07.743874+0000","last_peered":"2024-04-23T18:15:07.743874+0000","last_clean":"2024-04-23T18:15:07.743874+0000","last_became_active":"2024-04-23T18:13:05.025034+0000","last_became_peered":"2024-04-23T18:13:05.025034+0000","last_unstale":"2024-04-23T18:15:07.743874+0000","last_undegraded":"2024-04-23T18:15:07.743874+0000","last_fullsized":"2024-04-23T18:15:07.743874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:45.739777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459741+0000","last_change":"2024-04-23T18:13:05.521873+0000","last_active":"2024-04-23T18:15:07.459741+0000","last_peered":"2024-04-23T18:15:07.459741+0000","last_clean":"2024-04-23T18:15:07.459741+0000","last_became_active":"2024-04-23T18:13:05.521553+0000","last_became_peered":"2024-04-23T18:13:05.521553+0000","last_unstale":"2024-04-23T18:15:07.459741+0000","last_undegraded":"2024-04-23T18:15:07.459741+0000","last_fullsized":"2024-04-23T18:15:07.459741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:54.421055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467700+0000","last_change":"2024-04-23T18:13:05.022703+0000","last_active":"2024-04-23T18:15:07.467700+0000","last_peered":"2024-04-23T18:15:07.467700+0000","last_clean":"2024-04-23T18:15:07.467700+0000","last_became_active":"2024-04-23T18:13:05.022392+0000","last_became_peered":"2024-04-23T18:13:05.022392+0000","last_unstale":"2024-04-23T18:15:07.467700+0000","last_undegraded":"2024-04-23T18:15:07.467700+0000","last_fullsized":"2024-04-23T18:15:07.467700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:11.720181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366070+0000","last_change":"2024-04-23T18:13:05.527354+0000","last_active":"2024-04-23T18:15:07.366070+0000","last_peered":"2024-04-23T18:15:07.366070+0000","last_clean":"2024-04-23T18:15:07.366070+0000","last_became_active":"2024-04-23T18:13:05.527036+0000","last_became_peered":"2024-04-23T18:13:05.527036+0000","last_unstale":"2024-04-23T18:15:07.366070+0000","last_undegraded":"2024-04-23T18:15:07.366070+0000","last_fullsized":"2024-04-23T18:15:07.366070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:53.729013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470986+0000","last_change":"2024-04-23T18:13:05.024862+0000","last_active":"2024-04-23T18:15:07.470986+0000","last_peered":"2024-04-23T18:15:07.470986+0000","last_clean":"2024-04-23T18:15:07.470986+0000","last_became_active":"2024-04-23T18:13:05.024534+0000","last_became_peered":"2024-04-23T18:13:05.024534+0000","last_unstale":"2024-04-23T18:15:07.470986+0000","last_undegraded":"2024-04-23T18:15:07.470986+0000","last_fullsized":"2024-04-23T18:15:07.470986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:34:53.221062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619758+0000","last_change":"2024-04-23T18:13:05.531827+0000","last_active":"2024-04-23T18:15:07.619758+0000","last_peered":"2024-04-23T18:15:07.619758+0000","last_clean":"2024-04-23T18:15:07.619758+0000","last_became_active":"2024-04-23T18:13:05.531131+0000","last_became_peered":"2024-04-23T18:13:05.531131+0000","last_unstale":"2024-04-23T18:15:07.619758+0000","last_undegraded":"2024-04-23T18:15:07.619758+0000","last_fullsized":"2024-04-23T18:15:07.619758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:50:53.947792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367926+0000","last_change":"2024-04-23T18:13:05.019033+0000","last_active":"2024-04-23T18:15:07.367926+0000","last_peered":"2024-04-23T18:15:07.367926+0000","last_clean":"2024-04-23T18:15:07.367926+0000","last_became_active":"2024-04-23T18:13:05.018720+0000","last_became_peered":"2024-04-23T18:13:05.018720+0000","last_unstale":"2024-04-23T18:15:07.367926+0000","last_undegraded":"2024-04-23T18:15:07.367926+0000","last_fullsized":"2024-04-23T18:15:07.367926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:45.822812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131914+0000","last_change":"2024-04-23T18:13:05.531219+0000","last_active":"2024-04-23T18:15:08.131914+0000","last_peered":"2024-04-23T18:15:08.131914+0000","last_clean":"2024-04-23T18:15:08.131914+0000","last_became_active":"2024-04-23T18:13:05.530875+0000","last_became_peered":"2024-04-23T18:13:05.530875+0000","last_unstale":"2024-04-23T18:15:08.131914+0000","last_undegraded":"2024-04-23T18:15:08.131914+0000","last_fullsized":"2024-04-23T18:15:08.131914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:56.130180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.468168+0000","last_change":"2024-04-23T18:13:05.028688+0000","last_active":"2024-04-23T18:15:07.468168+0000","last_peered":"2024-04-23T18:15:07.468168+0000","last_clean":"2024-04-23T18:15:07.468168+0000","last_became_active":"2024-04-23T18:13:05.028372+0000","last_became_peered":"2024-04-23T18:13:05.028372+0000","last_unstale":"2024-04-23T18:15:07.468168+0000","last_undegraded":"2024-04-23T18:15:07.468168+0000","last_fullsized":"2024-04-23T18:15:07.468168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:18.801243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619569+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.619569+0000","last_peered":"2024-04-23T18:15:07.619569+0000","last_clean":"2024-04-23T18:15:07.619569+0000","last_became_active":"2024-04-23T18:13:05.531204+0000","last_became_peered":"2024-04-23T18:13:05.531204+0000","last_unstale":"2024-04-23T18:15:07.619569+0000","last_undegraded":"2024-04-23T18:15:07.619569+0000","last_fullsized":"2024-04-23T18:15:07.619569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:50.284151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.231237+0000","last_change":"2024-04-23T18:13:05.021280+0000","last_active":"2024-04-23T18:15:08.231237+0000","last_peered":"2024-04-23T18:15:08.231237+0000","last_clean":"2024-04-23T18:15:08.231237+0000","last_became_active":"2024-04-23T18:13:05.020938+0000","last_became_peered":"2024-04-23T18:13:05.020938+0000","last_unstale":"2024-04-23T18:15:08.231237+0000","last_undegraded":"2024-04-23T18:15:08.231237+0000","last_fullsized":"2024-04-23T18:15:08.231237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:38.416775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63985,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":187,"num_write":36,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460234+0000","last_change":"2024-04-23T18:13:05.527802+0000","last_active":"2024-04-23T18:15:07.460234+0000","last_peered":"2024-04-23T18:15:07.460234+0000","last_clean":"2024-04-23T18:15:07.460234+0000","last_became_active":"2024-04-23T18:13:05.527508+0000","last_became_peered":"2024-04-23T18:13:05.527508+0000","last_unstale":"2024-04-23T18:15:07.460234+0000","last_undegraded":"2024-04-23T18:15:07.460234+0000","last_fullsized":"2024-04-23T18:15:07.460234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:53.385412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371185+0000","last_change":"2024-04-23T18:13:05.016952+0000","last_active":"2024-04-23T18:15:07.371185+0000","last_peered":"2024-04-23T18:15:07.371185+0000","last_clean":"2024-04-23T18:15:07.371185+0000","last_became_active":"2024-04-23T18:13:05.016611+0000","last_became_peered":"2024-04-23T18:13:05.016611+0000","last_unstale":"2024-04-23T18:15:07.371185+0000","last_undegraded":"2024-04-23T18:15:07.371185+0000","last_fullsized":"2024-04-23T18:15:07.371185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:50.510285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743914+0000","last_change":"2024-04-23T18:13:05.526396+0000","last_active":"2024-04-23T18:15:07.743914+0000","last_peered":"2024-04-23T18:15:07.743914+0000","last_clean":"2024-04-23T18:15:07.743914+0000","last_became_active":"2024-04-23T18:13:05.526085+0000","last_became_peered":"2024-04-23T18:13:05.526085+0000","last_unstale":"2024-04-23T18:15:07.743914+0000","last_undegraded":"2024-04-23T18:15:07.743914+0000","last_fullsized":"2024-04-23T18:15:07.743914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:40:19.212867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467867+0000","last_change":"2024-04-23T18:13:05.025665+0000","last_active":"2024-04-23T18:15:07.467867+0000","last_peered":"2024-04-23T18:15:07.467867+0000","last_clean":"2024-04-23T18:15:07.467867+0000","last_became_active":"2024-04-23T18:13:05.025316+0000","last_became_peered":"2024-04-23T18:13:05.025316+0000","last_unstale":"2024-04-23T18:15:07.467867+0000","last_undegraded":"2024-04-23T18:15:07.467867+0000","last_fullsized":"2024-04-23T18:15:07.467867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:11.581013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368083+0000","last_change":"2024-04-23T18:13:05.531579+0000","last_active":"2024-04-23T18:15:07.368083+0000","last_peered":"2024-04-23T18:15:07.368083+0000","last_clean":"2024-04-23T18:15:07.368083+0000","last_became_active":"2024-04-23T18:13:05.531265+0000","last_became_peered":"2024-04-23T18:13:05.531265+0000","last_unstale":"2024-04-23T18:15:07.368083+0000","last_undegraded":"2024-04-23T18:15:07.368083+0000","last_fullsized":"2024-04-23T18:15:07.368083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:39.791068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":149,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232046+0000","last_change":"2024-04-23T18:13:05.015411+0000","last_active":"2024-04-23T18:15:08.232046+0000","last_peered":"2024-04-23T18:15:08.232046+0000","last_clean":"2024-04-23T18:15:08.232046+0000","last_became_active":"2024-04-23T18:13:05.015041+0000","last_became_peered":"2024-04-23T18:13:05.015041+0000","last_unstale":"2024-04-23T18:15:08.232046+0000","last_undegraded":"2024-04-23T18:15:08.232046+0000","last_fullsized":"2024-04-23T18:15:08.232046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:14.513180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367523+0000","last_change":"2024-04-23T18:13:05.532819+0000","last_active":"2024-04-23T18:15:07.367523+0000","last_peered":"2024-04-23T18:15:07.367523+0000","last_clean":"2024-04-23T18:15:07.367523+0000","last_became_active":"2024-04-23T18:13:05.532364+0000","last_became_peered":"2024-04-23T18:13:05.532364+0000","last_unstale":"2024-04-23T18:15:07.367523+0000","last_undegraded":"2024-04-23T18:15:07.367523+0000","last_fullsized":"2024-04-23T18:15:07.367523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:41:49.126644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459987+0000","last_change":"2024-04-23T18:13:05.028327+0000","last_active":"2024-04-23T18:15:07.459987+0000","last_peered":"2024-04-23T18:15:07.459987+0000","last_clean":"2024-04-23T18:15:07.459987+0000","last_became_active":"2024-04-23T18:13:05.027578+0000","last_became_peered":"2024-04-23T18:13:05.027578+0000","last_unstale":"2024-04-23T18:15:07.459987+0000","last_undegraded":"2024-04-23T18:15:07.459987+0000","last_fullsized":"2024-04-23T18:15:07.459987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:04.717366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467708+0000","last_change":"2024-04-23T18:13:05.536804+0000","last_active":"2024-04-23T18:15:07.467708+0000","last_peered":"2024-04-23T18:15:07.467708+0000","last_clean":"2024-04-23T18:15:07.467708+0000","last_became_active":"2024-04-23T18:13:05.536329+0000","last_became_peered":"2024-04-23T18:13:05.536329+0000","last_unstale":"2024-04-23T18:15:07.467708+0000","last_undegraded":"2024-04-23T18:15:07.467708+0000","last_fullsized":"2024-04-23T18:15:07.467708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:10.612627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460312+0000","last_change":"2024-04-23T18:13:05.024568+0000","last_active":"2024-04-23T18:15:07.460312+0000","last_peered":"2024-04-23T18:15:07.460312+0000","last_clean":"2024-04-23T18:15:07.460312+0000","last_became_active":"2024-04-23T18:13:05.024210+0000","last_became_peered":"2024-04-23T18:13:05.024210+0000","last_unstale":"2024-04-23T18:15:07.460312+0000","last_undegraded":"2024-04-23T18:15:07.460312+0000","last_fullsized":"2024-04-23T18:15:07.460312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:16.232440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428679+0000","last_change":"2024-04-23T18:13:57.292094+0000","last_active":"2024-04-23T18:15:07.428679+0000","last_peered":"2024-04-23T18:15:07.428679+0000","last_clean":"2024-04-23T18:15:07.428679+0000","last_became_active":"2024-04-23T18:13:57.291761+0000","last_became_peered":"2024-04-23T18:13:57.291761+0000","last_unstale":"2024-04-23T18:15:07.428679+0000","last_undegraded":"2024-04-23T18:15:07.428679+0000","last_fullsized":"2024-04-23T18:15:07.428679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:38:31.544598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.366373+0000","last_change":"2024-04-23T18:13:05.031255+0000","last_active":"2024-04-23T18:15:07.366373+0000","last_peered":"2024-04-23T18:15:07.366373+0000","last_clean":"2024-04-23T18:15:07.366373+0000","last_became_active":"2024-04-23T18:13:05.030231+0000","last_became_peered":"2024-04-23T18:13:05.030231+0000","last_unstale":"2024-04-23T18:15:07.366373+0000","last_undegraded":"2024-04-23T18:15:07.366373+0000","last_fullsized":"2024-04-23T18:15:07.366373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:23.090449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470956+0000","last_change":"2024-04-23T18:13:05.525872+0000","last_active":"2024-04-23T18:15:07.470956+0000","last_peered":"2024-04-23T18:15:07.470956+0000","last_clean":"2024-04-23T18:15:07.470956+0000","last_became_active":"2024-04-23T18:13:05.525514+0000","last_became_peered":"2024-04-23T18:13:05.525514+0000","last_unstale":"2024-04-23T18:15:07.470956+0000","last_undegraded":"2024-04-23T18:15:07.470956+0000","last_fullsized":"2024-04-23T18:15:07.470956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:25.584664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.368585+0000","last_change":"2024-04-23T18:13:05.017981+0000","last_active":"2024-04-23T18:15:07.368585+0000","last_peered":"2024-04-23T18:15:07.368585+0000","last_clean":"2024-04-23T18:15:07.368585+0000","last_became_active":"2024-04-23T18:13:05.017706+0000","last_became_peered":"2024-04-23T18:13:05.017706+0000","last_unstale":"2024-04-23T18:15:07.368585+0000","last_undegraded":"2024-04-23T18:15:07.368585+0000","last_fullsized":"2024-04-23T18:15:07.368585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:22.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.470804+0000","last_change":"2024-04-23T18:13:57.202859+0000","last_active":"2024-04-23T18:15:07.470804+0000","last_peered":"2024-04-23T18:15:07.470804+0000","last_clean":"2024-04-23T18:15:07.470804+0000","last_became_active":"2024-04-23T18:13:57.202395+0000","last_became_peered":"2024-04-23T18:13:57.202395+0000","last_unstale":"2024-04-23T18:15:07.470804+0000","last_undegraded":"2024-04-23T18:15:07.470804+0000","last_fullsized":"2024-04-23T18:15:07.470804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:17.871010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467943+0000","last_change":"2024-04-23T18:13:05.529513+0000","last_active":"2024-04-23T18:15:07.467943+0000","last_peered":"2024-04-23T18:15:07.467943+0000","last_clean":"2024-04-23T18:15:07.467943+0000","last_became_active":"2024-04-23T18:13:05.529179+0000","last_became_peered":"2024-04-23T18:13:05.529179+0000","last_unstale":"2024-04-23T18:15:07.467943+0000","last_undegraded":"2024-04-23T18:15:07.467943+0000","last_fullsized":"2024-04-23T18:15:07.467943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:54.058925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743768+0000","last_change":"2024-04-23T18:13:05.010261+0000","last_active":"2024-04-23T18:15:07.743768+0000","last_peered":"2024-04-23T18:15:07.743768+0000","last_clean":"2024-04-23T18:15:07.743768+0000","last_became_active":"2024-04-23T18:13:05.009892+0000","last_became_peered":"2024-04-23T18:13:05.009892+0000","last_unstale":"2024-04-23T18:15:07.743768+0000","last_undegraded":"2024-04-23T18:15:07.743768+0000","last_fullsized":"2024-04-23T18:15:07.743768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:21.648016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460210+0000","last_change":"2024-04-23T18:13:05.535126+0000","last_active":"2024-04-23T18:15:07.460210+0000","last_peered":"2024-04-23T18:15:07.460210+0000","last_clean":"2024-04-23T18:15:07.460210+0000","last_became_active":"2024-04-23T18:13:05.528536+0000","last_became_peered":"2024-04-23T18:13:05.528536+0000","last_unstale":"2024-04-23T18:15:07.460210+0000","last_undegraded":"2024-04-23T18:15:07.460210+0000","last_fullsized":"2024-04-23T18:15:07.460210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:30:14.232588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232092+0000","last_change":"2024-04-23T18:13:05.023825+0000","last_active":"2024-04-23T18:15:08.232092+0000","last_peered":"2024-04-23T18:15:08.232092+0000","last_clean":"2024-04-23T18:15:08.232092+0000","last_became_active":"2024-04-23T18:13:05.023221+0000","last_became_peered":"2024-04-23T18:13:05.023221+0000","last_unstale":"2024-04-23T18:15:08.232092+0000","last_undegraded":"2024-04-23T18:15:08.232092+0000","last_fullsized":"2024-04-23T18:15:08.232092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:01.366010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367609+0000","last_change":"2024-04-23T18:11:51.498541+0000","last_active":"2024-04-23T18:15:07.367609+0000","last_peered":"2024-04-23T18:15:07.367609+0000","last_clean":"2024-04-23T18:15:07.367609+0000","last_became_active":"2024-04-23T18:11:51.498125+0000","last_became_peered":"2024-04-23T18:11:51.498125+0000","last_unstale":"2024-04-23T18:15:07.367609+0000","last_undegraded":"2024-04-23T18:15:07.367609+0000","last_fullsized":"2024-04-23T18:15:07.367609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:08:22.810545+0000","last_clean_scrub_stamp":"2024-04-23T18:08:22.810545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:48.152125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743700+0000","last_change":"2024-04-23T18:13:05.529621+0000","last_active":"2024-04-23T18:15:07.743700+0000","last_peered":"2024-04-23T18:15:07.743700+0000","last_clean":"2024-04-23T18:15:07.743700+0000","last_became_active":"2024-04-23T18:13:05.529290+0000","last_became_peered":"2024-04-23T18:13:05.529290+0000","last_unstale":"2024-04-23T18:15:07.743700+0000","last_undegraded":"2024-04-23T18:15:07.743700+0000","last_fullsized":"2024-04-23T18:15:07.743700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:50.782308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367785+0000","last_change":"2024-04-23T18:13:05.023669+0000","last_active":"2024-04-23T18:15:07.367785+0000","last_peered":"2024-04-23T18:15:07.367785+0000","last_clean":"2024-04-23T18:15:07.367785+0000","last_became_active":"2024-04-23T18:13:05.023004+0000","last_became_peered":"2024-04-23T18:13:05.023004+0000","last_unstale":"2024-04-23T18:15:07.367785+0000","last_undegraded":"2024-04-23T18:15:07.367785+0000","last_fullsized":"2024-04-23T18:15:07.367785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:17.228929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460339+0000","last_change":"2024-04-23T18:13:05.028492+0000","last_active":"2024-04-23T18:15:07.460339+0000","last_peered":"2024-04-23T18:15:07.460339+0000","last_clean":"2024-04-23T18:15:07.460339+0000","last_became_active":"2024-04-23T18:13:05.028164+0000","last_became_peered":"2024-04-23T18:13:05.028164+0000","last_unstale":"2024-04-23T18:15:07.460339+0000","last_undegraded":"2024-04-23T18:15:07.460339+0000","last_fullsized":"2024-04-23T18:15:07.460339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:34.842223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744119+0000","last_change":"2024-04-23T18:13:05.522626+0000","last_active":"2024-04-23T18:15:07.744119+0000","last_peered":"2024-04-23T18:15:07.744119+0000","last_clean":"2024-04-23T18:15:07.744119+0000","last_became_active":"2024-04-23T18:13:05.522302+0000","last_became_peered":"2024-04-23T18:13:05.522302+0000","last_unstale":"2024-04-23T18:15:07.744119+0000","last_undegraded":"2024-04-23T18:15:07.744119+0000","last_fullsized":"2024-04-23T18:15:07.744119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:22.311506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'45","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.582411+0000","last_change":"2024-04-23T18:13:05.031046+0000","last_active":"2024-04-23T18:15:07.582411+0000","last_peered":"2024-04-23T18:15:07.582411+0000","last_clean":"2024-04-23T18:15:07.582411+0000","last_became_active":"2024-04-23T18:13:05.030260+0000","last_became_peered":"2024-04-23T18:13:05.030260+0000","last_unstale":"2024-04-23T18:15:07.582411+0000","last_undegraded":"2024-04-23T18:15:07.582411+0000","last_fullsized":"2024-04-23T18:15:07.582411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:15:03.095164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460514+0000","last_change":"2024-04-23T18:13:05.537852+0000","last_active":"2024-04-23T18:15:07.460514+0000","last_peered":"2024-04-23T18:15:07.460514+0000","last_clean":"2024-04-23T18:15:07.460514+0000","last_became_active":"2024-04-23T18:13:05.532507+0000","last_became_peered":"2024-04-23T18:13:05.532507+0000","last_unstale":"2024-04-23T18:15:07.460514+0000","last_undegraded":"2024-04-23T18:15:07.460514+0000","last_fullsized":"2024-04-23T18:15:07.460514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:22.077154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.232315+0000","last_change":"2024-04-23T18:13:05.017678+0000","last_active":"2024-04-23T18:15:08.232315+0000","last_peered":"2024-04-23T18:15:08.232315+0000","last_clean":"2024-04-23T18:15:08.232315+0000","last_became_active":"2024-04-23T18:13:05.017308+0000","last_became_peered":"2024-04-23T18:13:05.017308+0000","last_unstale":"2024-04-23T18:15:08.232315+0000","last_undegraded":"2024-04-23T18:15:08.232315+0000","last_fullsized":"2024-04-23T18:15:08.232315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:32:20.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131760+0000","last_change":"2024-04-23T18:13:05.532465+0000","last_active":"2024-04-23T18:15:08.131760+0000","last_peered":"2024-04-23T18:15:08.131760+0000","last_clean":"2024-04-23T18:15:08.131760+0000","last_became_active":"2024-04-23T18:13:05.532150+0000","last_became_peered":"2024-04-23T18:13:05.532150+0000","last_unstale":"2024-04-23T18:15:08.131760+0000","last_undegraded":"2024-04-23T18:15:08.131760+0000","last_fullsized":"2024-04-23T18:15:08.131760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:51.328678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619167+0000","last_change":"2024-04-23T18:13:05.020527+0000","last_active":"2024-04-23T18:15:07.619167+0000","last_peered":"2024-04-23T18:15:07.619167+0000","last_clean":"2024-04-23T18:15:07.619167+0000","last_became_active":"2024-04-23T18:13:05.020156+0000","last_became_peered":"2024-04-23T18:13:05.020156+0000","last_unstale":"2024-04-23T18:15:07.619167+0000","last_undegraded":"2024-04-23T18:15:07.619167+0000","last_fullsized":"2024-04-23T18:15:07.619167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:24:56.250939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459984+0000","last_change":"2024-04-23T18:13:05.528571+0000","last_active":"2024-04-23T18:15:07.459984+0000","last_peered":"2024-04-23T18:15:07.459984+0000","last_clean":"2024-04-23T18:15:07.459984+0000","last_became_active":"2024-04-23T18:13:05.526397+0000","last_became_peered":"2024-04-23T18:13:05.526397+0000","last_unstale":"2024-04-23T18:15:07.459984+0000","last_undegraded":"2024-04-23T18:15:07.459984+0000","last_fullsized":"2024-04-23T18:15:07.459984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:42:58.724047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459981+0000","last_change":"2024-04-23T18:13:05.026590+0000","last_active":"2024-04-23T18:15:07.459981+0000","last_peered":"2024-04-23T18:15:07.459981+0000","last_clean":"2024-04-23T18:15:07.459981+0000","last_became_active":"2024-04-23T18:13:05.026286+0000","last_became_peered":"2024-04-23T18:13:05.026286+0000","last_unstale":"2024-04-23T18:15:07.459981+0000","last_undegraded":"2024-04-23T18:15:07.459981+0000","last_fullsized":"2024-04-23T18:15:07.459981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:52.559540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428476+0000","last_change":"2024-04-23T18:13:05.525660+0000","last_active":"2024-04-23T18:15:07.428476+0000","last_peered":"2024-04-23T18:15:07.428476+0000","last_clean":"2024-04-23T18:15:07.428476+0000","last_became_active":"2024-04-23T18:13:05.525337+0000","last_became_peered":"2024-04-23T18:13:05.525337+0000","last_unstale":"2024-04-23T18:15:07.428476+0000","last_undegraded":"2024-04-23T18:15:07.428476+0000","last_fullsized":"2024-04-23T18:15:07.428476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:39.378527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428685+0000","last_change":"2024-04-23T18:13:57.207521+0000","last_active":"2024-04-23T18:15:07.428685+0000","last_peered":"2024-04-23T18:15:07.428685+0000","last_clean":"2024-04-23T18:15:07.428685+0000","last_became_active":"2024-04-23T18:13:57.201507+0000","last_became_peered":"2024-04-23T18:13:57.201507+0000","last_unstale":"2024-04-23T18:15:07.428685+0000","last_undegraded":"2024-04-23T18:15:07.428685+0000","last_fullsized":"2024-04-23T18:15:07.428685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:35.254723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460384+0000","last_change":"2024-04-23T18:13:05.537079+0000","last_active":"2024-04-23T18:15:07.460384+0000","last_peered":"2024-04-23T18:15:07.460384+0000","last_clean":"2024-04-23T18:15:07.460384+0000","last_became_active":"2024-04-23T18:13:05.531784+0000","last_became_peered":"2024-04-23T18:13:05.531784+0000","last_unstale":"2024-04-23T18:15:07.460384+0000","last_undegraded":"2024-04-23T18:15:07.460384+0000","last_fullsized":"2024-04-23T18:15:07.460384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:16.588951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460148+0000","last_change":"2024-04-23T18:13:05.029664+0000","last_active":"2024-04-23T18:15:07.460148+0000","last_peered":"2024-04-23T18:15:07.460148+0000","last_clean":"2024-04-23T18:15:07.460148+0000","last_became_active":"2024-04-23T18:13:05.029227+0000","last_became_peered":"2024-04-23T18:13:05.029227+0000","last_unstale":"2024-04-23T18:15:07.460148+0000","last_undegraded":"2024-04-23T18:15:07.460148+0000","last_fullsized":"2024-04-23T18:15:07.460148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:36.188701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367860+0000","last_change":"2024-04-23T18:13:05.533378+0000","last_active":"2024-04-23T18:15:07.367860+0000","last_peered":"2024-04-23T18:15:07.367860+0000","last_clean":"2024-04-23T18:15:07.367860+0000","last_became_active":"2024-04-23T18:13:05.532987+0000","last_became_peered":"2024-04-23T18:13:05.532987+0000","last_unstale":"2024-04-23T18:15:07.367860+0000","last_undegraded":"2024-04-23T18:15:07.367860+0000","last_fullsized":"2024-04-23T18:15:07.367860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:19:45.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619248+0000","last_change":"2024-04-23T18:13:57.206276+0000","last_active":"2024-04-23T18:15:07.619248+0000","last_peered":"2024-04-23T18:15:07.619248+0000","last_clean":"2024-04-23T18:15:07.619248+0000","last_became_active":"2024-04-23T18:13:57.202137+0000","last_became_peered":"2024-04-23T18:13:57.202137+0000","last_unstale":"2024-04-23T18:15:07.619248+0000","last_undegraded":"2024-04-23T18:15:07.619248+0000","last_fullsized":"2024-04-23T18:15:07.619248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:12.998331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367944+0000","last_change":"2024-04-23T18:13:05.534420+0000","last_active":"2024-04-23T18:15:07.367944+0000","last_peered":"2024-04-23T18:15:07.367944+0000","last_clean":"2024-04-23T18:15:07.367944+0000","last_became_active":"2024-04-23T18:13:05.533975+0000","last_became_peered":"2024-04-23T18:13:05.533975+0000","last_unstale":"2024-04-23T18:15:07.367944+0000","last_undegraded":"2024-04-23T18:15:07.367944+0000","last_fullsized":"2024-04-23T18:15:07.367944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:22.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.577849+0000","last_change":"2024-04-23T18:13:05.027975+0000","last_active":"2024-04-23T18:15:07.577849+0000","last_peered":"2024-04-23T18:15:07.577849+0000","last_clean":"2024-04-23T18:15:07.577849+0000","last_became_active":"2024-04-23T18:13:05.027629+0000","last_became_peered":"2024-04-23T18:13:05.027629+0000","last_unstale":"2024-04-23T18:15:07.577849+0000","last_undegraded":"2024-04-23T18:15:07.577849+0000","last_fullsized":"2024-04-23T18:15:07.577849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.018771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367840+0000","last_change":"2024-04-23T18:13:05.530934+0000","last_active":"2024-04-23T18:15:07.367840+0000","last_peered":"2024-04-23T18:15:07.367840+0000","last_clean":"2024-04-23T18:15:07.367840+0000","last_became_active":"2024-04-23T18:13:05.530375+0000","last_became_peered":"2024-04-23T18:13:05.530375+0000","last_unstale":"2024-04-23T18:15:07.367840+0000","last_undegraded":"2024-04-23T18:15:07.367840+0000","last_fullsized":"2024-04-23T18:15:07.367840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:01:31.449429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.744110+0000","last_change":"2024-04-23T18:13:05.018883+0000","last_active":"2024-04-23T18:15:07.744110+0000","last_peered":"2024-04-23T18:15:07.744110+0000","last_clean":"2024-04-23T18:15:07.744110+0000","last_became_active":"2024-04-23T18:13:05.018502+0000","last_became_peered":"2024-04-23T18:13:05.018502+0000","last_unstale":"2024-04-23T18:15:07.744110+0000","last_undegraded":"2024-04-23T18:15:07.744110+0000","last_fullsized":"2024-04-23T18:15:07.744110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:03.996299+0000","last_clean_scrub_stamp":"2024-04-23T18:13:03.996299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:56:19.513016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.367824+0000","last_change":"2024-04-23T18:13:05.532373+0000","last_active":"2024-04-23T18:15:07.367824+0000","last_peered":"2024-04-23T18:15:07.367824+0000","last_clean":"2024-04-23T18:15:07.367824+0000","last_became_active":"2024-04-23T18:13:05.532028+0000","last_became_peered":"2024-04-23T18:13:05.532028+0000","last_unstale":"2024-04-23T18:15:07.367824+0000","last_undegraded":"2024-04-23T18:15:07.367824+0000","last_fullsized":"2024-04-23T18:15:07.367824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:51.881754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428772+0000","last_change":"2024-04-23T18:13:05.529549+0000","last_active":"2024-04-23T18:15:07.428772+0000","last_peered":"2024-04-23T18:15:07.428772+0000","last_clean":"2024-04-23T18:15:07.428772+0000","last_became_active":"2024-04-23T18:13:05.529257+0000","last_became_peered":"2024-04-23T18:13:05.529257+0000","last_unstale":"2024-04-23T18:15:07.428772+0000","last_undegraded":"2024-04-23T18:15:07.428772+0000","last_fullsized":"2024-04-23T18:15:07.428772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:04:22.333869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371128+0000","last_change":"2024-04-23T18:13:05.534560+0000","last_active":"2024-04-23T18:15:07.371128+0000","last_peered":"2024-04-23T18:15:07.371128+0000","last_clean":"2024-04-23T18:15:07.371128+0000","last_became_active":"2024-04-23T18:13:05.534189+0000","last_became_peered":"2024-04-23T18:13:05.534189+0000","last_unstale":"2024-04-23T18:15:07.371128+0000","last_undegraded":"2024-04-23T18:15:07.371128+0000","last_fullsized":"2024-04-23T18:15:07.371128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:41.000390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371055+0000","last_change":"2024-04-23T18:13:05.528674+0000","last_active":"2024-04-23T18:15:07.371055+0000","last_peered":"2024-04-23T18:15:07.371055+0000","last_clean":"2024-04-23T18:15:07.371055+0000","last_became_active":"2024-04-23T18:13:05.528345+0000","last_became_peered":"2024-04-23T18:13:05.528345+0000","last_unstale":"2024-04-23T18:15:07.371055+0000","last_undegraded":"2024-04-23T18:15:07.371055+0000","last_fullsized":"2024-04-23T18:15:07.371055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:01.991923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:08.131355+0000","last_change":"2024-04-23T18:13:05.538175+0000","last_active":"2024-04-23T18:15:08.131355+0000","last_peered":"2024-04-23T18:15:08.131355+0000","last_clean":"2024-04-23T18:15:08.131355+0000","last_became_active":"2024-04-23T18:13:05.537823+0000","last_became_peered":"2024-04-23T18:13:05.537823+0000","last_unstale":"2024-04-23T18:15:08.131355+0000","last_undegraded":"2024-04-23T18:15:08.131355+0000","last_fullsized":"2024-04-23T18:15:08.131355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743963+0000","last_change":"2024-04-23T18:13:05.529691+0000","last_active":"2024-04-23T18:15:07.743963+0000","last_peered":"2024-04-23T18:15:07.743963+0000","last_clean":"2024-04-23T18:15:07.743963+0000","last_became_active":"2024-04-23T18:13:05.529351+0000","last_became_peered":"2024-04-23T18:13:05.529351+0000","last_unstale":"2024-04-23T18:15:07.743963+0000","last_undegraded":"2024-04-23T18:15:07.743963+0000","last_fullsized":"2024-04-23T18:15:07.743963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:48:04.108260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428702+0000","last_change":"2024-04-23T18:13:05.528910+0000","last_active":"2024-04-23T18:15:07.428702+0000","last_peered":"2024-04-23T18:15:07.428702+0000","last_clean":"2024-04-23T18:15:07.428702+0000","last_became_active":"2024-04-23T18:13:05.528522+0000","last_became_peered":"2024-04-23T18:13:05.528522+0000","last_unstale":"2024-04-23T18:15:07.428702+0000","last_undegraded":"2024-04-23T18:15:07.428702+0000","last_fullsized":"2024-04-23T18:15:07.428702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:00.721477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.371506+0000","last_change":"2024-04-23T18:13:05.522575+0000","last_active":"2024-04-23T18:15:07.371506+0000","last_peered":"2024-04-23T18:15:07.371506+0000","last_clean":"2024-04-23T18:15:07.371506+0000","last_became_active":"2024-04-23T18:13:05.522025+0000","last_became_peered":"2024-04-23T18:13:05.522025+0000","last_unstale":"2024-04-23T18:15:07.371506+0000","last_undegraded":"2024-04-23T18:15:07.371506+0000","last_fullsized":"2024-04-23T18:15:07.371506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:53:02.155634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460394+0000","last_change":"2024-04-23T18:13:05.537540+0000","last_active":"2024-04-23T18:15:07.460394+0000","last_peered":"2024-04-23T18:15:07.460394+0000","last_clean":"2024-04-23T18:15:07.460394+0000","last_became_active":"2024-04-23T18:13:05.535896+0000","last_became_peered":"2024-04-23T18:13:05.535896+0000","last_unstale":"2024-04-23T18:15:07.460394+0000","last_undegraded":"2024-04-23T18:15:07.460394+0000","last_fullsized":"2024-04-23T18:15:07.460394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:07.362253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.467697+0000","last_change":"2024-04-23T18:13:05.525534+0000","last_active":"2024-04-23T18:15:07.467697+0000","last_peered":"2024-04-23T18:15:07.467697+0000","last_clean":"2024-04-23T18:15:07.467697+0000","last_became_active":"2024-04-23T18:13:05.525226+0000","last_became_peered":"2024-04-23T18:13:05.525226+0000","last_unstale":"2024-04-23T18:15:07.467697+0000","last_undegraded":"2024-04-23T18:15:07.467697+0000","last_fullsized":"2024-04-23T18:15:07.467697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:09.339001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.459881+0000","last_change":"2024-04-23T18:13:05.527576+0000","last_active":"2024-04-23T18:15:07.459881+0000","last_peered":"2024-04-23T18:15:07.459881+0000","last_clean":"2024-04-23T18:15:07.459881+0000","last_became_active":"2024-04-23T18:13:05.527250+0000","last_became_peered":"2024-04-23T18:13:05.527250+0000","last_unstale":"2024-04-23T18:15:07.459881+0000","last_undegraded":"2024-04-23T18:15:07.459881+0000","last_fullsized":"2024-04-23T18:15:07.459881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:28.733524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.619351+0000","last_change":"2024-04-23T18:13:05.534036+0000","last_active":"2024-04-23T18:15:07.619351+0000","last_peered":"2024-04-23T18:15:07.619351+0000","last_clean":"2024-04-23T18:15:07.619351+0000","last_became_active":"2024-04-23T18:13:05.529354+0000","last_became_peered":"2024-04-23T18:13:05.529354+0000","last_unstale":"2024-04-23T18:15:07.619351+0000","last_undegraded":"2024-04-23T18:15:07.619351+0000","last_fullsized":"2024-04-23T18:15:07.619351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:02:34.875550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.743931+0000","last_change":"2024-04-23T18:13:05.524999+0000","last_active":"2024-04-23T18:15:07.743931+0000","last_peered":"2024-04-23T18:15:07.743931+0000","last_clean":"2024-04-23T18:15:07.743931+0000","last_became_active":"2024-04-23T18:13:05.524637+0000","last_became_peered":"2024-04-23T18:13:05.524637+0000","last_unstale":"2024-04-23T18:15:07.743931+0000","last_undegraded":"2024-04-23T18:15:07.743931+0000","last_fullsized":"2024-04-23T18:15:07.743931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:47.508548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428525+0000","last_change":"2024-04-23T18:13:57.199427+0000","last_active":"2024-04-23T18:15:07.428525+0000","last_peered":"2024-04-23T18:15:07.428525+0000","last_clean":"2024-04-23T18:15:07.428525+0000","last_became_active":"2024-04-23T18:13:57.199001+0000","last_became_peered":"2024-04-23T18:13:57.199001+0000","last_unstale":"2024-04-23T18:15:07.428525+0000","last_undegraded":"2024-04-23T18:15:07.428525+0000","last_fullsized":"2024-04-23T18:15:07.428525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:13:04.997957+0000","last_clean_scrub_stamp":"2024-04-23T18:13:04.997957+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:14.788961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108174,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594174,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":840,"num_read_kb":527,"num_write":289,"num_write_kb":1435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1164,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1782522,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":469494,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1257092,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":600140,"data_compressed":9316,"data_compressed_allocated":12288,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":1062561,"data_compressed":38944,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":533479,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":470390,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":533723,"data_compressed":2886,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":994246,"data_compressed":28188,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1193704,"data_compressed":45100,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":197695,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:15:13.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/3138662063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 msgr2=0x7f9b4410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/3138662063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b4410ee40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009ca0 tx=0x7f9b3c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/3138662063 shutdown_connections 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/3138662063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b4410ee40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/3138662063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b44102b50 0x7f9b44106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/3138662063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 0x7f9b44102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/3138662063 >> 172.21.15.32:0/3138662063 conn(0x7f9b440fda40 msgr2=0x7f9b440ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/3138662063 shutdown_connections 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.571+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/3138662063 wait complete. 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 Processor -- start 2024-04-23T18:15:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 -- start start 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 0x7f9b44198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b4419d000 0x7f9b4419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441121e0 con 0x7f9b44107270 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44112060 con 0x7f9b441021a0 2024-04-23T18:15:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4bbd8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44112360 con 0x7f9b4419d000 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41044/0 (socket says 172.21.15.32:41044) 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 -- 172.21.15.32:0/86678562 learned_addr learned my addr 172.21.15.32:0/86678562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49974640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 0x7f9b44198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b4a175640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b4419d000 0x7f9b4419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 -- 172.21.15.32:0/86678562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b4419d000 msgr2=0x7f9b4419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b4419d000 0x7f9b4419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 -- 172.21.15.32:0/86678562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 msgr2=0x7f9b44198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 0x7f9b44198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.572+0000 7f9b49173640 1 -- 172.21.15.32:0/86678562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4419daf0 con 0x7f9b44107270 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b49173640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400d740 tx=0x7f9b3400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34017070 con 0x7f9b44107270 2024-04-23T18:15:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3400de50 con 0x7f9b44107270 2024-04-23T18:15:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4419dde0 con 0x7f9b44107270 2024-04-23T18:15:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34004ce0 con 0x7f9b44107270 2024-04-23T18:15:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.573+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a5580 con 0x7f9b44107270 2024-04-23T18:15:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.574+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b44069250 con 0x7f9b44107270 2024-04-23T18:15:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.575+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f9b34004e80 con 0x7f9b44107270 2024-04-23T18:15:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.576+0000 7f9b3affd640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 0x7f9b300a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.576+0000 7f9b49974640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 0x7f9b300a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.576+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f9b340d06c0 con 0x7f9b44107270 2024-04-23T18:15:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.576+0000 7f9b49974640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 0x7f9b300a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009bd0 tx=0x7f9b40005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.579+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b3409cc40 con 0x7f9b44107270 2024-04-23T18:15:13.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:13 smithi064 ceph-mon[33062]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:13.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:13 smithi032 ceph-mon[29221]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:13.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.805+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b44102b50 con 0x7f9b44107270 2024-04-23T18:15:13.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.805+0000 7f9b3affd640 1 -- 172.21.15.32:0/86678562 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11851 (secure 0 0 0) 0x7f9b3409c540 con 0x7f9b44107270 2024-04-23T18:15:13.799 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:13.800 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:05.923544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:15:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-23T18:15:13.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 msgr2=0x7f9b300a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 0x7f9b300a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009bd0 tx=0x7f9b40005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:13.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 msgr2=0x7f9b44198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400d740 tx=0x7f9b3400dc00 comp rx=0 tx=0).stop 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 shutdown_connections 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b4419d000 0x7f9b4419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9b300a3db0 0x7f9b300a6270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b44107270 0x7f9b44198c80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 --2- 172.21.15.32:0/86678562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441021a0 0x7f9b44198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 >> 172.21.15.32:0/86678562 conn(0x7f9b440fda40 msgr2=0x7f9b440ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 shutdown_connections 2024-04-23T18:15:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.807+0000 7f9b4bbd8640 1 -- 172.21.15.32:0/86678562 wait complete. 2024-04-23T18:15:13.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:13.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:13 smithi040 ceph-mon[32626]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.920+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/781756959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe28103a90 msgr2=0x7fbe2810bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.920+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/781756959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe28103a90 0x7fbe2810bf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009900 tx=0x7fbe1c02a480 comp rx=0 tx=0).stop 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/781756959 shutdown_connections 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/781756959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe2810c540 0x7fbe2810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/781756959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe28103a90 0x7fbe2810bf70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/781756959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103170 0x7fbe28103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/781756959 >> 172.21.15.32:0/781756959 conn(0x7fbe28078060 msgr2=0x7fbe28078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/781756959 shutdown_connections 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/781756959 wait complete. 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 Processor -- start 2024-04-23T18:15:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.921+0000 7fbe2cb6f640 1 -- start start 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe28103170 0x7fbe2819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103a90 0x7fbe2819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28112180 con 0x7fbe2810c540 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28112000 con 0x7fbe28103170 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28112300 con 0x7fbe28103a90 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41062/0 (socket says 172.21.15.32:41062) 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 -- 172.21.15.32:0/226297039 learned_addr learned my addr 172.21.15.32:0/226297039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2659c640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe28103170 0x7fbe2819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe25d9b640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103a90 0x7fbe2819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 -- 172.21.15.32:0/226297039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103a90 msgr2=0x7fbe2819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103a90 0x7fbe2819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 -- 172.21.15.32:0/226297039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe28103170 msgr2=0x7fbe2819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe28103170 0x7fbe2819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 -- 172.21.15.32:0/226297039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe28198d70 con 0x7fbe2810c540 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe26d9d640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000eaa0 tx=0x7fbe2000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2000cdb0 con 0x7fbe2810c540 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe20005020 con 0x7fbe2810c540 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe28199060 con 0x7fbe2810c540 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.922+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe200116d0 con 0x7fbe2810c540 2024-04-23T18:15:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.923+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe281a5300 con 0x7fbe2810c540 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.924+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2806ae60 con 0x7fbe2810c540 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.924+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fbe200051d0 con 0x7fbe2810c540 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.925+0000 7fbe177fe640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 0x7fbe100a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.925+0000 7fbe2659c640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 0x7fbe100a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.926+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fbe200d0b50 con 0x7fbe2810c540 2024-04-23T18:15:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.926+0000 7fbe2659c640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 0x7fbe100a6270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009bd0 tx=0x7fbe18005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:13.930+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe20016030 con 0x7fbe2810c540 2024-04-23T18:15:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.117+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe28107630 con 0x7fbe2810c540 2024-04-23T18:15:14.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.119+0000 7fbe177fe640 1 -- 172.21.15.32:0/226297039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22910 (secure 0 0 0) 0x7fbe2009d070 con 0x7fbe2810c540 2024-04-23T18:15:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:14.113 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":89,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:15:07.361332+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 msgr2=0x7fbe100a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 0x7fbe100a6270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009bd0 tx=0x7fbe18005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 msgr2=0x7fbe281985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000eaa0 tx=0x7fbe2000ef60 comp rx=0 tx=0).stop 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 shutdown_connections 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbe100a3db0 0x7fbe100a6270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe2810c540 0x7fbe281985f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe28103a90 0x7fbe2819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 --2- 172.21.15.32:0/226297039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe28103170 0x7fbe2819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 >> 172.21.15.32:0/226297039 conn(0x7fbe28078060 msgr2=0x7fbe280ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 shutdown_connections 2024-04-23T18:15:14.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.121+0000 7fbe2cb6f640 1 -- 172.21.15.32:0/226297039 wait complete. 2024-04-23T18:15:14.126 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:14.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- 172.21.15.32:0/738700703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770106160 msgr2=0x7fc77010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 --2- 172.21.15.32:0/738700703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770106160 0x7fc77010dd30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc768009ca0 tx=0x7fc76802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- 172.21.15.32:0/738700703 shutdown_connections 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 --2- 172.21.15.32:0/738700703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770106160 0x7fc77010dd30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 --2- 172.21.15.32:0/738700703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770101a40 0x7fc770105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 --2- 172.21.15.32:0/738700703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 0x7fc770101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- 172.21.15.32:0/738700703 >> 172.21.15.32:0/738700703 conn(0x7fc770078000 msgr2=0x7fc770078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- 172.21.15.32:0/738700703 shutdown_connections 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- 172.21.15.32:0/738700703 wait complete. 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 Processor -- start 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.238+0000 7fc7760c2640 1 -- start start 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 0x7fc7701a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770106160 0x7fc7701a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701121e0 con 0x7fc77019a700 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc770112060 con 0x7fc770101090 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc7760c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc770112360 con 0x7fc770106160 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41086/0 (socket says 172.21.15.32:41086) 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 -- 172.21.15.32:0/2183630462 learned_addr learned my addr 172.21.15.32:0/2183630462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76f7fe640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 0x7fc7701a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76effd640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770106160 0x7fc7701a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 -- 172.21.15.32:0/2183630462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770106160 msgr2=0x7fc7701a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770106160 0x7fc7701a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 -- 172.21.15.32:0/2183630462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 msgr2=0x7fc7701a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 0x7fc7701a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 -- 172.21.15.32:0/2183630462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77019b300 con 0x7fc77019a700 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.239+0000 7fc76ffff640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc768005cc0 tx=0x7fc768031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.240+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768003b70 con 0x7fc77019a700 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.240+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77019b590 con 0x7fc77019a700 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.240+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7680318f0 con 0x7fc77019a700 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.240+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76802f950 con 0x7fc77019a700 2024-04-23T18:15:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.240+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701a7780 con 0x7fc77019a700 2024-04-23T18:15:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.241+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc770102560 con 0x7fc77019a700 2024-04-23T18:15:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.242+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fc76802fb50 con 0x7fc77019a700 2024-04-23T18:15:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.242+0000 7fc76cff9640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 0x7fc7580a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.242+0000 7fc76f7fe640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 0x7fc7580a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.242+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc7680f1b30 con 0x7fc77019a700 2024-04-23T18:15:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.243+0000 7fc76f7fe640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 0x7fc7580a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc76000bbe0 tx=0x7fc76000b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.247+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7680be1c0 con 0x7fc77019a700 2024-04-23T18:15:14.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.434+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc770101a40 con 0x7fc77019a700 2024-04-23T18:15:14.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.434+0000 7fc76cff9640 1 -- 172.21.15.32:0/2183630462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc7680bdac0 con 0x7fc77019a700 2024-04-23T18:15:14.429 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 msgr2=0x7fc7580a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 0x7fc7580a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc76000bbe0 tx=0x7fc76000b390 comp rx=0 tx=0).stop 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 msgr2=0x7fc77019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc768005cc0 tx=0x7fc768031700 comp rx=0 tx=0).stop 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 shutdown_connections 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc7580a3db0 0x7fc7580a6270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77019a700 0x7fc77019ab80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc770106160 0x7fc7701a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 --2- 172.21.15.32:0/2183630462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770101090 0x7fc7701a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 >> 172.21.15.32:0/2183630462 conn(0x7fc770078000 msgr2=0x7fc770107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 shutdown_connections 2024-04-23T18:15:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.436+0000 7fc7760c2640 1 -- 172.21.15.32:0/2183630462 wait complete. 2024-04-23T18:15:14.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:15:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/2724251618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e01010a0 msgr2=0x7fa3e0101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/2724251618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e01010a0 0x7fa3e0101480 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009900 tx=0x7fa3d002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/2724251618 shutdown_connections 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/2724251618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e0106170 0x7fa3e010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/2724251618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e0101a50 0x7fa3e0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/2724251618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e01010a0 0x7fa3e0101480 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/2724251618 >> 172.21.15.32:0/2724251618 conn(0x7fa3e0077f70 msgr2=0x7fa3e0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/2724251618 shutdown_connections 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.553+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/2724251618 wait complete. 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 Processor -- start 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 -- start start 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e01010a0 0x7fa3e01a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 0x7fa3e019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e01110b0 con 0x7fa3e0106170 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0110f30 con 0x7fa3e019a640 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3e4b6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0111230 con 0x7fa3e01010a0 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3ddd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3ddd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41098/0 (socket says 172.21.15.32:41098) 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.554+0000 7fa3ddd9b640 1 -- 172.21.15.32:0/3252123475 learned_addr learned my addr 172.21.15.32:0/3252123475 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3de59c640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e01010a0 0x7fa3e01a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ded9d640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 0x7fa3e019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 -- 172.21.15.32:0/3252123475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e01010a0 msgr2=0x7fa3e01a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e01010a0 0x7fa3e01a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 -- 172.21.15.32:0/3252123475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 msgr2=0x7fa3e019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 0x7fa3e019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 -- 172.21.15.32:0/3252123475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e019b160 con 0x7fa3e0106170 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ded9d640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 0x7fa3e019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3ddd9b640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c970 tx=0x7fa3d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4014070 con 0x7fa3e0106170 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e019b450 con 0x7fa3e0106170 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3d4004440 con 0x7fa3e0106170 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4003e50 con 0x7fa3e0106170 2024-04-23T18:15:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.555+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e01a7650 con 0x7fa3e0106170 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.556+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e0069250 con 0x7fa3e0106170 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.557+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fa3d4017070 con 0x7fa3e0106170 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.558+0000 7fa3cf7fe640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 0x7fa3c80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.558+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa3d40d0590 con 0x7fa3e0106170 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.558+0000 7fa3de59c640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 0x7fa3c80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.558+0000 7fa3de59c640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 0x7fa3c80a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0031440 tx=0x7fa3d00095f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.562+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3d401e030 con 0x7fa3e0106170 2024-04-23T18:15:14.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:14 smithi064 ceph-mon[33062]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:14 smithi064 ceph-mon[33062]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:14.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:14 smithi064 ceph-mon[33062]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:14 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/86678562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:14 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/226297039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.750+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3e0104170 con 0x7fa3e0106170 2024-04-23T18:15:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.750+0000 7fa3cf7fe640 1 -- 172.21.15.32:0/3252123475 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fa3d400de20 con 0x7fa3e0106170 2024-04-23T18:15:14.744 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:15:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 msgr2=0x7fa3c80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 0x7fa3c80a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0031440 tx=0x7fa3d00095f0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 msgr2=0x7fa3e01a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c970 tx=0x7fa3d400ce30 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 shutdown_connections 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3e019a640 0x7fa3e019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3c80a3f50 0x7fa3c80a6410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e0106170 0x7fa3e01a0980 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 --2- 172.21.15.32:0/3252123475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3e01010a0 0x7fa3e01a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 >> 172.21.15.32:0/3252123475 conn(0x7fa3e0077f70 msgr2=0x7fa3e00fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 shutdown_connections 2024-04-23T18:15:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.752+0000 7fa3e4b6f640 1 -- 172.21.15.32:0/3252123475 wait complete. 2024-04-23T18:15:14.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:14 smithi032 ceph-mon[29221]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:14 smithi032 ceph-mon[29221]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:14.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:14 smithi032 ceph-mon[29221]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:14 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/86678562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:14 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/226297039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- 172.21.15.32:0/1310444822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 msgr2=0x7f583410eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 --2- 172.21.15.32:0/1310444822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f583410eef0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f582c009cd0 tx=0x7f582c02a8e0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- 172.21.15.32:0/1310444822 shutdown_connections 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 --2- 172.21.15.32:0/1310444822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f583410eef0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 --2- 172.21.15.32:0/1310444822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 0x7f5834103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 --2- 172.21.15.32:0/1310444822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 0x7f5834102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- 172.21.15.32:0/1310444822 >> 172.21.15.32:0/1310444822 conn(0x7f58340fda10 msgr2=0x7f58340ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- 172.21.15.32:0/1310444822 shutdown_connections 2024-04-23T18:15:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- 172.21.15.32:0/1310444822 wait complete. 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 Processor -- start 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.869+0000 7f583a807640 1 -- start start 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 0x7f5834074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 0x7f5834074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5834112290 con 0x7f58341044f0 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5834112110 con 0x7f5834102270 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5834112410 con 0x7f5834103b50 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41116/0 (socket says 172.21.15.32:41116) 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 -- 172.21.15.32:0/933967101 learned_addr learned my addr 172.21.15.32:0/933967101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5833fff640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 0x7f5834074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f58337fe640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 0x7f5834074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 -- 172.21.15.32:0/933967101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 msgr2=0x7f5834074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 0x7f5834074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 -- 172.21.15.32:0/933967101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 msgr2=0x7f5834074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 0x7f5834074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 -- 172.21.15.32:0/933967101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58341a6c50 con 0x7f58341044f0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f5838da4640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f582c005cc0 tx=0x7f582c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c004450 con 0x7f58341044f0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f582c031c90 con 0x7f58341044f0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.870+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58341a6ee0 con 0x7f58341044f0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.871+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c03b400 con 0x7f58341044f0 2024-04-23T18:15:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.871+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58341a7450 con 0x7f58341044f0 2024-04-23T18:15:14.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:14 smithi040 ceph-mon[32626]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:14 smithi040 ceph-mon[32626]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:14.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:14 smithi040 ceph-mon[32626]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:14.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:14 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/86678562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:14 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/226297039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.871+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5834069250 con 0x7f58341044f0 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.873+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f582c004680 con 0x7f58341044f0 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.873+0000 7f58317fa640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 0x7f581c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.874+0000 7f5833fff640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 0x7f581c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.874+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f582c02e030 con 0x7f58341044f0 2024-04-23T18:15:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.874+0000 7f5833fff640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 0x7f581c0a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f582400bbe0 tx=0x7f582400b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:14.878+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f582c0bded0 con 0x7f58341044f0 2024-04-23T18:15:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.064+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f583410a570 con 0x7f58341044f0 2024-04-23T18:15:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.064+0000 7f58317fa640 1 -- 172.21.15.32:0/933967101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f582c0098b0 con 0x7f58341044f0 2024-04-23T18:15:15.059 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:15:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.066+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 msgr2=0x7f581c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.066+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 0x7f581c0a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f582400bbe0 tx=0x7f582400b390 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.066+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 msgr2=0x7f5834074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.066+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f582c005cc0 tx=0x7f582c003c20 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 shutdown_connections 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f581c0a3e60 0x7f581c0a6320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58341044f0 0x7f5834074cd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5834103b50 0x7f5834074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 --2- 172.21.15.32:0/933967101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5834102270 0x7f5834074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 >> 172.21.15.32:0/933967101 conn(0x7f58340fda10 msgr2=0x7f58340fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 shutdown_connections 2024-04-23T18:15:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.067+0000 7f583a807640 1 -- 172.21.15.32:0/933967101 wait complete. 2024-04-23T18:15:15.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 -- 172.21.15.32:0/1356567383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c1010a0 msgr2=0x7fb05c101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1356567383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c1010a0 0x7fb05c101500 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb044009900 tx=0x7fb04402a410 comp rx=0 tx=0).stop 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 -- 172.21.15.32:0/1356567383 shutdown_connections 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1356567383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c101a40 0x7fb05c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1356567383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c1010a0 0x7fb05c101500 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1356567383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c106ed0 0x7fb05c1072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 -- 172.21.15.32:0/1356567383 >> 172.21.15.32:0/1356567383 conn(0x7fb05c077c40 msgr2=0x7fb05c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 -- 172.21.15.32:0/1356567383 shutdown_connections 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.180+0000 7fb060f6f640 1 -- 172.21.15.32:0/1356567383 wait complete. 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 Processor -- start 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 -- start start 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c1010a0 0x7fb05c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c106ed0 0x7fb05c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1110a0 con 0x7fb05c101a40 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c110f20 con 0x7fb05c106ed0 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb060f6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c111220 con 0x7fb05c1010a0 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb053fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb053fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41138/0 (socket says 172.21.15.32:41138) 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb053fff640 1 -- 172.21.15.32:0/1230534978 learned_addr learned my addr 172.21.15.32:0/1230534978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb05a59c640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c1010a0 0x7fb05c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.181+0000 7fb05ad9d640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c106ed0 0x7fb05c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 -- 172.21.15.32:0/1230534978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c1010a0 msgr2=0x7fb05c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c1010a0 0x7fb05c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 -- 172.21.15.32:0/1230534978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c106ed0 msgr2=0x7fb05c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c106ed0 0x7fb05c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 -- 172.21.15.32:0/1230534978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c19d750 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb053fff640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb044005940 tx=0x7fb044004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04403f030 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c19d9e0 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb044003aa0 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb044035070 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.182+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c1a52d0 con 0x7fb05c101a40 2024-04-23T18:15:15.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.183+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c068e00 con 0x7fb05c101a40 2024-04-23T18:15:15.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.184+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7fb044042030 con 0x7fb05c101a40 2024-04-23T18:15:15.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.185+0000 7fb0537fe640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 0x7fb0480a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.185+0000 7fb05a59c640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 0x7fb0480a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.185+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fb0440f0ff0 con 0x7fb05c101a40 2024-04-23T18:15:15.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.185+0000 7fb05a59c640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 0x7fb0480a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb04c005fd0 tx=0x7fb04c00b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.189+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0440bcec0 con 0x7fb05c101a40 2024-04-23T18:15:15.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.414+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb05c1a54a0 con 0x7fb05c101a40 2024-04-23T18:15:15.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.415+0000 7fb0537fe640 1 -- 172.21.15.32:0/1230534978 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11851 (secure 0 0 0) 0x7fb0440c11c0 con 0x7fb05c101a40 2024-04-23T18:15:15.408 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:15.409 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:05.923544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:15:15.409 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.416+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 msgr2=0x7fb0480a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.416+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 0x7fb0480a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb04c005fd0 tx=0x7fb04c00b390 comp rx=0 tx=0).stop 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 msgr2=0x7fb05c198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb044005940 tx=0x7fb044004270 comp rx=0 tx=0).stop 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 shutdown_connections 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05c106ed0 0x7fb05c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb0480a3db0 0x7fb0480a6270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb05c101a40 0x7fb05c198ca0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 --2- 172.21.15.32:0/1230534978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb05c1010a0 0x7fb05c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 >> 172.21.15.32:0/1230534978 conn(0x7fb05c077c40 msgr2=0x7fb05c0694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 shutdown_connections 2024-04-23T18:15:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.417+0000 7fb060f6f640 1 -- 172.21.15.32:0/1230534978 wait complete. 2024-04-23T18:15:15.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 -- 172.21.15.32:0/3766388049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130101a50 msgr2=0x7f0130105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 --2- 172.21.15.32:0/3766388049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130101a50 0x7f0130105a90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0118009900 tx=0x7f011802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 -- 172.21.15.32:0/3766388049 shutdown_connections 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 --2- 172.21.15.32:0/3766388049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 0x7f013010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 --2- 172.21.15.32:0/3766388049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130101a50 0x7f0130105a90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 --2- 172.21.15.32:0/3766388049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 -- 172.21.15.32:0/3766388049 >> 172.21.15.32:0/3766388049 conn(0x7f0130077f50 msgr2=0x7f0130078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 -- 172.21.15.32:0/3766388049 shutdown_connections 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.530+0000 7f0135dbb640 1 -- 172.21.15.32:0/3766388049 wait complete. 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 Processor -- start 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 -- start start 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 0x7f01301999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0130111090 con 0x7f0130199f40 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0130110f10 con 0x7f0130106170 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f0135dbb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0130111210 con 0x7f01301010a0 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f012ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f012ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41152/0 (socket says 172.21.15.32:41152) 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f012ffff640 1 -- 172.21.15.32:0/20082469 learned_addr learned my addr 172.21.15.32:0/20082469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.531+0000 7f012effd640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 0x7f01301999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012f7fe640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 -- 172.21.15.32:0/20082469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 msgr2=0x7f0130199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 -- 172.21.15.32:0/20082469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 msgr2=0x7f01301999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 0x7f01301999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 -- 172.21.15.32:0/20082469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01301a4cc0 con 0x7f0130199f40 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012f7fe640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012ffff640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f012800cce0 tx=0x7f012800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012800f450 con 0x7f0130199f40 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012800faa0 con 0x7f0130199f40 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01301a4fb0 con 0x7f0130199f40 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128018b70 con 0x7f0130199f40 2024-04-23T18:15:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.532+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01301a5480 con 0x7f0130199f40 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.533+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0130069250 con 0x7f0130199f40 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.534+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f0128018d10 con 0x7f0130199f40 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.535+0000 7f012cff9640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 0x7f011c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.535+0000 7f012f7fe640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 0x7f011c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.535+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f01280d0d60 con 0x7f0130199f40 2024-04-23T18:15:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.535+0000 7f012f7fe640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 0x7f011c0a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f012000abe0 tx=0x7f012000a5c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.540+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0128014030 con 0x7f0130199f40 2024-04-23T18:15:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:15 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2183630462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:15 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3252123475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:15 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/933967101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.728+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01301093a0 con 0x7f0130199f40 2024-04-23T18:15:15.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.729+0000 7f012cff9640 1 -- 172.21.15.32:0/20082469 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22910 (secure 0 0 0) 0x7f012809d2e0 con 0x7f0130199f40 2024-04-23T18:15:15.723 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:15.724 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":89,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:15:07.361332+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 msgr2=0x7f011c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 0x7f011c0a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f012000abe0 tx=0x7f012000a5c0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 msgr2=0x7f01301a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f012800cce0 tx=0x7f012800a620 comp rx=0 tx=0).stop 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 shutdown_connections 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f011c0a3db0 0x7f011c0a6270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0130199f40 0x7f01301a44f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0130106170 0x7f01301999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 --2- 172.21.15.32:0/20082469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01301010a0 0x7f0130199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 >> 172.21.15.32:0/20082469 conn(0x7f0130077f50 msgr2=0x7f01300feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 shutdown_connections 2024-04-23T18:15:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.731+0000 7f0135dbb640 1 -- 172.21.15.32:0/20082469 wait complete. 2024-04-23T18:15:15.735 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:15 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2183630462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.735 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:15 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3252123475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.736 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:15 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/933967101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.737 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 384471 2024-04-23T18:15:15.737 DEBUG:tasks.quiescer:Yielding 2024-04-23T18:15:15.737 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:15:15.748 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:15:15.748 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:15:15.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-23T18:15:15.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:15.782 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:15.797 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-23T18:15:15.797 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:3300,172.21.15.40:3300,172.21.15.64:3300:/volumes/qa/sv_1/d17f95c9-36ff-4b3f-b9ea-0f18c979081d 340G 0 340G 0% /tmp/cephfs 2024-04-23T18:15:15.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-23T18:15:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 -- 172.21.15.32:0/83639953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8102b50 msgr2=0x7f5fc8106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 --2- 172.21.15.32:0/83639953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8102b50 0x7f5fc8106b90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009900 tx=0x7f5fbc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 -- 172.21.15.32:0/83639953 shutdown_connections 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 --2- 172.21.15.32:0/83639953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 --2- 172.21.15.32:0/83639953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8102b50 0x7f5fc8106b90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 --2- 172.21.15.32:0/83639953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 0x7f5fc8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.859+0000 7f5fcee10640 1 -- 172.21.15.32:0/83639953 >> 172.21.15.32:0/83639953 conn(0x7f5fc80fda40 msgr2=0x7f5fc80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- 172.21.15.32:0/83639953 shutdown_connections 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- 172.21.15.32:0/83639953 wait complete. 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 Processor -- start 2024-04-23T18:15:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- start start 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 0x7f5fc819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 0x7f5fc8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112190 con 0x7f5fc8198590 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112010 con 0x7f5fc81021a0 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.860+0000 7f5fcee10640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112310 con 0x7f5fc8107270 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fcd3ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 0x7f5fc8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36388/0 (socket says 172.21.15.32:36388) 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 -- 172.21.15.32:0/1743914030 learned_addr learned my addr 172.21.15.32:0/1743914030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fccbac640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 0x7f5fc819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 -- 172.21.15.32:0/1743914030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 msgr2=0x7f5fc819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 0x7f5fc819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 -- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 msgr2=0x7f5fc8198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 0x7f5fc8198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 -- 172.21.15.32:0/1743914030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc81990b0 con 0x7f5fc8107270 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fcd3ad640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 0x7f5fc8198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc7fff640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc032fd0 tx=0x7f5fbc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc03f030 con 0x7f5fc8107270 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc8199340 con 0x7f5fc8107270 2024-04-23T18:15:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fbc004490 con 0x7f5fc8107270 2024-04-23T18:15:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.861+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc035070 con 0x7f5fc8107270 2024-04-23T18:15:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.862+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc81a5590 con 0x7f5fc8107270 2024-04-23T18:15:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.863+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc8068e00 con 0x7f5fc8107270 2024-04-23T18:15:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.864+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f5fbc042030 con 0x7f5fc8107270 2024-04-23T18:15:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.865+0000 7f5fc5ffb640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 0x7f5fb00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.865+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f5fbc0f1190 con 0x7f5fc8107270 2024-04-23T18:15:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.865+0000 7f5fccbac640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 0x7f5fb00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.865+0000 7f5fccbac640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 0x7f5fb00a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80058c0 tx=0x7f5fb800a2d0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:15.869+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fbc0bd060 con 0x7f5fc8107270 2024-04-23T18:15:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:15 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2183630462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:15 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3252123475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:15 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/933967101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:15:15.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-23T18:15:15.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-23T18:15:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.096+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc8199ac0 con 0x7f5fc8107270 2024-04-23T18:15:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.097+0000 7f5fc5ffb640 1 -- 172.21.15.32:0/1743914030 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11851 (secure 0 0 0) 0x7f5fbc0c1360 con 0x7f5fc8107270 2024-04-23T18:15:16.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:16.091 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:05.923544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:15:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 msgr2=0x7f5fb00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 0x7f5fb00a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80058c0 tx=0x7f5fb800a2d0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 msgr2=0x7f5fc819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc032fd0 tx=0x7f5fbc004270 comp rx=0 tx=0).stop 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 shutdown_connections 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5fb00a3db0 0x7f5fb00a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fc8198590 0x7f5fc8198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fc8107270 0x7f5fc819e920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 --2- 172.21.15.32:0/1743914030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc81021a0 0x7f5fc819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 >> 172.21.15.32:0/1743914030 conn(0x7f5fc80fda40 msgr2=0x7f5fc80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 shutdown_connections 2024-04-23T18:15:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.099+0000 7f5fcee10640 1 -- 172.21.15.32:0/1743914030 wait complete. 2024-04-23T18:15:16.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:13:05.375853+0000', 'modified': '2024-04-23T18:15:05.923544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24551, 'mds_1': 24515, 'mds_2': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/669117639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 669117639}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 669117639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/1068324383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1068324383}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1068324383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6837/1625538187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 1625538187}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 1625538187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.64:6839/1242971440', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 1242971440}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 1242971440}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6837/1154751886', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 1154751886}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 1154751886}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2479383965', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2479383965}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2479383965}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24551, 'qdb_cluster': [24551, 24499, 24515]} 2024-04-23T18:15:16.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:15:16.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:15:16.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 -- 172.21.15.32:0/3019641005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2390100f30 msgr2=0x7f2390101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 --2- 172.21.15.32:0/3019641005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2390100f30 0x7f2390101390 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2378009930 tx=0x7f237802a420 comp rx=0 tx=0).stop 2024-04-23T18:15:16.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 -- 172.21.15.32:0/3019641005 shutdown_connections 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 --2- 172.21.15.32:0/3019641005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23901018d0 0x7f239010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 --2- 172.21.15.32:0/3019641005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2390100f30 0x7f2390101390 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 --2- 172.21.15.32:0/3019641005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 0x7f2390107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 -- 172.21.15.32:0/3019641005 >> 172.21.15.32:0/3019641005 conn(0x7f2390076f20 msgr2=0x7f2390077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 -- 172.21.15.32:0/3019641005 shutdown_connections 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.216+0000 7f23948e8640 1 -- 172.21.15.32:0/3019641005 wait complete. 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 Processor -- start 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 -- start start 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390100f30 0x7f23901a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 0x7f239019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2390111180 con 0x7f23901018d0 2024-04-23T18:15:16.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2390111000 con 0x7f2390100f30 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.217+0000 7f23948e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2390111300 con 0x7f2390106f50 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41170/0 (socket says 172.21.15.32:41170) 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 -- 172.21.15.32:0/4263073518 learned_addr learned my addr 172.21.15.32:0/4263073518 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f238ed9d640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 0x7f239019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f238e59c640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390100f30 0x7f23901a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 -- 172.21.15.32:0/4263073518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 msgr2=0x7f239019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 0x7f239019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 -- 172.21.15.32:0/4263073518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390100f30 msgr2=0x7f23901a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390100f30 0x7f23901a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 -- 172.21.15.32:0/4263073518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239019af70 con 0x7f23901018d0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f2387fff640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2378005ab0 tx=0x7f2378003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f23877fe640 1 -- 172.21.15.32:0/4263073518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237803f070 con 0x7f23901018d0 2024-04-23T18:15:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239019b200 con 0x7f23901018d0 2024-04-23T18:15:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23901a7480 con 0x7f23901018d0 2024-04-23T18:15:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f23877fe640 1 -- 172.21.15.32:0/4263073518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2378031b50 con 0x7f23901018d0 2024-04-23T18:15:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.218+0000 7f23877fe640 1 -- 172.21.15.32:0/4263073518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378035030 con 0x7f23901018d0 2024-04-23T18:15:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 Processor -- start 2024-04-23T18:15:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 -- start start 2024-04-23T18:15:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf061db0 0x5618bf062190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf06cdd0 0x5618bf06f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618befc7c50 con 0x5618bf062760 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618bef64830 con 0x5618bf06cdd0 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f2397602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5618befc3c10 con 0x5618bf061db0 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41184/0 (socket says 172.21.15.32:41184) 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 -- 172.21.15.32:0/3364609924 learned_addr learned my addr 172.21.15.32:0/3364609924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238ed9d640 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf061db0 0x5618bf062190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 -- 172.21.15.32:0/3364609924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf061db0 msgr2=0x5618bf062190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf061db0 0x5618bf062190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 -- 172.21.15.32:0/3364609924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf06cdd0 msgr2=0x5618bf06f220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf06cdd0 0x5618bf06f220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.221+0000 7f238e59c640 1 -- 172.21.15.32:0/3364609924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618becda2d0 con 0x5618bf062760 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f238e59c640 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2380009ca0 tx=0x7f2380011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed121e8fcce44af0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f23877fe640 1 -- 172.21.15.32:0/4263073518 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f2378042070 con 0x7f23901018d0 2024-04-23T18:15:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f2384ff9640 1 -- 172.21.15.32:0/3364609924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238001e070 con 0x5618bf062760 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f2384ff9640 1 -- 172.21.15.32:0/3364609924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2380019ae0 con 0x5618bf062760 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f2384ff9640 1 -- 172.21.15.32:0/3364609924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238000e960 con 0x5618bf062760 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.222+0000 7f23877fe640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 0x7f237c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f238ed9d640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 0x7f237c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f23877fe640 1 -- 172.21.15.32:0/4263073518 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f23780f0c00 con 0x7f23901018d0 2024-04-23T18:15:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 -- 172.21.15.32:0/3364609924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 msgr2=0x5618bf06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2380009ca0 tx=0x7f2380011810 comp rx=0 tx=0).stop 2024-04-23T18:15:16.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f238ed9d640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 0x7f237c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2388005fd0 tx=0x7f23880096a0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 -- 172.21.15.32:0/3364609924 shutdown_connections 2024-04-23T18:15:16.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf06cdd0 0x5618bf06f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf062760 0x5618bf06c800 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 --2- 172.21.15.32:0/3364609924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf061db0 0x5618bf062190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 -- 172.21.15.32:0/3364609924 >> 172.21.15.32:0/3364609924 conn(0x5618bf05e070 msgr2=0x5618bf05e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 -- 172.21.15.32:0/3364609924 shutdown_connections 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.223+0000 7f2397602740 1 -- 172.21.15.32:0/3364609924 wait complete. 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 -- start start 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 Processor -- start 2024-04-23T18:15:16.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf069d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf06a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618bf06fd40 con 0x5618bf06cdd0 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618bef62470 con 0x5618bf061db0 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5618bf06fec0 con 0x5618bf062760 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2397602740 10 client.0 _ll_register_callbacks cb 0x5618bef87f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41186/0 (socket says 172.21.15.32:41186) 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 -- 172.21.15.32:0/3521408653 learned_addr learned my addr 172.21.15.32:0/3521408653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f2387fff640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf06a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238e59c640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf069d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 -- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 msgr2=0x5618bf06a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf06a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.224+0000 7f238ed9d640 1 -- 172.21.15.32:0/3521408653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 msgr2=0x5618bf069d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f238ed9d640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf069d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f238ed9d640 1 -- 172.21.15.32:0/3521408653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618beda46b0 con 0x5618bf06cdd0 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f238e59c640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf069d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f238ed9d640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2388012700 tx=0x7f2388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1700e169199a3c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388010a70 con 0x5618bf06cdd0 2024-04-23T18:15:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f2388010c10 con 0x5618bf06cdd0 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2380019ae0 con 0x5618bf06cdd0 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2388010e10 con 0x5618bf06cdd0 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2397602740 10 client.15507 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5618bf07c0a0 con 0x5618bf06cdd0 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2380019ae0 con 0x5618bf06cdd0 2024-04-23T18:15:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.225+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2388024070 con 0x5618bf06cdd0 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 10 client.15507 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 10 client.15507 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 20 client.15507 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf07fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 4 client.15507 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618befd5670 con 0x5618bf07e740 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f238e59c640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf07fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f2375ffb640 10 client.15507 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.226+0000 7f238e59c640 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf07fbb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40e8a5b2d00b8646 server_cookie=e8756ca7c88edf7f in_seq=0 out_seq=0 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2375ffb640 1 -- 172.21.15.32:0/3521408653 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618befd5670 con 0x5618bf07e740 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2375ffb640 10 client.15507 handle_command_reply: tid=0 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 client.15507 shutdown 2024-04-23T18:15:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 msgr2=0x5618bf06a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2388012700 tx=0x7f2388004a40 comp rx=0 tx=0).stop 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 shutdown_connections 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf07fbb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf06a780 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2375ffb640 0 client.15507 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 --2- 172.21.15.32:0/3521408653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf069d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 >> 172.21.15.32:0/3521408653 conn(0x5618bf05e070 msgr2=0x5618bf063ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 shutdown_connections 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.227+0000 7f2397602740 1 -- 172.21.15.32:0/3521408653 wait complete. 2024-04-23T18:15:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.228+0000 7f2397602740 20 client.15507 trim_cache size 0 max 16384 2024-04-23T18:15:16.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.264+0000 7f2397602740 1 Processor -- start 2024-04-23T18:15:16.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.264+0000 7f2397602740 1 -- start start 2024-04-23T18:15:16.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.264+0000 7f2397602740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf0643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.264+0000 7f2397602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf0685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2397602740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2397602740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618bf07def0 con 0x5618bf06cdd0 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2397602740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618bf07e5c0 con 0x5618bf061db0 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2397602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5618bf07ec90 con 0x5618bf062760 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41200/0 (socket says 172.21.15.32:41200) 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 -- 172.21.15.32:0/1114081051 learned_addr learned my addr 172.21.15.32:0/1114081051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238e59c640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf0643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2387fff640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf0685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 -- 172.21.15.32:0/1114081051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 msgr2=0x5618bf0685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf0685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 -- 172.21.15.32:0/1114081051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 msgr2=0x5618bf0643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf0643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 -- 172.21.15.32:0/1114081051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618bf068250 con 0x5618bf06cdd0 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2387fff640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf0685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f238ed9d640 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f23880266b0 tx=0x7f23880044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10265e164d3d0839 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2377fff640 1 -- 172.21.15.32:0/1114081051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388033030 con 0x5618bf06cdd0 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2377fff640 1 -- 172.21.15.32:0/1114081051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2388004780 con 0x5618bf06cdd0 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.265+0000 7f2377fff640 1 -- 172.21.15.32:0/1114081051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238801b460 con 0x5618bf06cdd0 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- 172.21.15.32:0/1114081051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 msgr2=0x5618bf068b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f23880266b0 tx=0x7f23880044d0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- 172.21.15.32:0/1114081051 shutdown_connections 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf068b10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf0685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 --2- 172.21.15.32:0/1114081051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf0643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- 172.21.15.32:0/1114081051 >> 172.21.15.32:0/1114081051 conn(0x5618bf05e070 msgr2=0x5618bf058e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- 172.21.15.32:0/1114081051 shutdown_connections 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- 172.21.15.32:0/1114081051 wait complete. 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.266+0000 7f2397602740 1 -- start start 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 Processor -- start 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf088110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf081500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618bf07fa30 con 0x5618bf06cdd0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618bf07f360 con 0x5618bf061db0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5618bf080100 con 0x5618bf062760 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2397602740 10 client.0 _ll_register_callbacks cb 0x7f2388003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41208/0 (socket says 172.21.15.32:41208) 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 -- 172.21.15.32:0/2106736440 learned_addr learned my addr 172.21.15.32:0/2106736440 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238e59c640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf088110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f2387fff640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf081500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 -- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 msgr2=0x5618bf081500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf081500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 -- 172.21.15.32:0/2106736440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 msgr2=0x5618bf088110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf088110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238ed9d640 1 -- 172.21.15.32:0/2106736440 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618bf0820d0 con 0x5618bf06cdd0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.267+0000 7f238e59c640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf088110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f238ed9d640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2388000f80 tx=0x7f2388005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd70612544606b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388033070 con 0x5618bf06cdd0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f2388004a40 con 0x5618bf06cdd0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2388004780 con 0x5618bf06cdd0 2024-04-23T18:15:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2388010c30 con 0x5618bf06cdd0 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 10 client.15513 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5618bf0e46f0 con 0x5618bf06cdd0 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2388004780 con 0x5618bf06cdd0 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2388024070 con 0x5618bf06cdd0 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 10 client.15513 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 10 client.15513 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:15:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 20 client.15513 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf0e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 4 client.15513 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:15:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.268+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 --> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] -- command(tid 0: {"prefix": "status"}) -- 0x5618befcf4e0 con 0x5618bf07e740 2024-04-23T18:15:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f238e59c640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf0e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f238e59c640 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf0e7970 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8cbca2b0b5c9b46c server_cookie=794686caa55e9c53 in_seq=0 out_seq=0 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2375ffb640 10 client.15513 ms_handle_connect on v2:172.21.15.64:6838/1242971440 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2375ffb640 1 -- 172.21.15.32:0/2106736440 <== mds.2 v2:172.21.15.64:6838/1242971440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5618befcf4e0 con 0x5618bf07e740 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2375ffb640 10 client.15513 handle_command_reply: tid=0 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 client.15513 shutdown 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 msgr2=0x5618bf081a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2388000f80 tx=0x7f2388005c90 comp rx=0 tx=0).stop 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 shutdown_connections 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:6838/1242971440,v1:172.21.15.64:6839/1242971440] conn(0x5618bf07e740 0x5618bf0e7970 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618bf06cdd0 0x5618bf081a40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5618bf062760 0x5618bf081500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2375ffb640 0 client.15513 ms_handle_reset on v2:172.21.15.64:6838/1242971440 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 --2- 172.21.15.32:0/2106736440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618bf061db0 0x5618bf088110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 >> 172.21.15.32:0/2106736440 conn(0x5618bf05e070 msgr2=0x5618bf068460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.269+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 shutdown_connections 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f2397602740 1 -- 172.21.15.32:0/2106736440 wait complete. 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f2397602740 20 client.15513 trim_cache size 0 max 16384 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24499, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 127.231556793, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 22, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 89, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 144.52204246400001 2024-04-23T18:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 msgr2=0x7f237c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 0x7f237c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2388005fd0 tx=0x7f23880096a0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 msgr2=0x7f23901a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2378005ab0 tx=0x7f2378003b90 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 shutdown_connections 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2390106f50 0x7f239019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f237c0a3db0 0x7f237c0a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23901018d0 0x7f23901a0b70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 --2- 172.21.15.32:0/4263073518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390100f30 0x7f23901a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 >> 172.21.15.32:0/4263073518 conn(0x7f2390076f20 msgr2=0x7f2390069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 shutdown_connections 2024-04-23T18:15:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.270+0000 7f23948e8640 1 -- 172.21.15.32:0/4263073518 wait complete. 2024-04-23T18:15:16.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 -- 172.21.15.32:0/4279365268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 msgr2=0x7f68d40698a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 --2- 172.21.15.32:0/4279365268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d40698a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009900 tx=0x7f68d002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 -- 172.21.15.32:0/4279365268 shutdown_connections 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 --2- 172.21.15.32:0/4279365268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 0x7f68d410ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 --2- 172.21.15.32:0/4279365268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68d4069e70 0x7f68d4106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 --2- 172.21.15.32:0/4279365268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d40698a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 -- 172.21.15.32:0/4279365268 >> 172.21.15.32:0/4279365268 conn(0x7f68d4076f20 msgr2=0x7f68d4077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 -- 172.21.15.32:0/4279365268 shutdown_connections 2024-04-23T18:15:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.389+0000 7f68db823640 1 -- 172.21.15.32:0/4279365268 wait complete. 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 Processor -- start 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 -- start start 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 0x7f68d419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68d4198670 0x7f68d4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4112290 con 0x7f68d40694c0 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4112110 con 0x7f68d41073a0 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68db823640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4112410 con 0x7f68d4198670 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41230/0 (socket says 172.21.15.32:41230) 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 -- 172.21.15.32:0/2303245232 learned_addr learned my addr 172.21.15.32:0/2303245232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 0x7f68d419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 -- 172.21.15.32:0/2303245232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68d4198670 msgr2=0x7f68d4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68d4198670 0x7f68d4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 -- 172.21.15.32:0/2303245232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 msgr2=0x7f68d419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 0x7f68d419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.390+0000 7f68d95bf640 1 -- 172.21.15.32:0/2303245232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d4199190 con 0x7f68d40694c0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005940 tx=0x7f68d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68ca7fc640 1 -- 172.21.15.32:0/2303245232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d003f030 con 0x7f68d40694c0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d4199420 con 0x7f68d40694c0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68ca7fc640 1 -- 172.21.15.32:0/2303245232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68d0003cb0 con 0x7f68d40694c0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68ca7fc640 1 -- 172.21.15.32:0/2303245232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0035070 con 0x7f68d40694c0 2024-04-23T18:15:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.391+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d41a55c0 con 0x7f68d40694c0 2024-04-23T18:15:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.393+0000 7f68ca7fc640 1 -- 172.21.15.32:0/2303245232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f68d0042030 con 0x7f68d40694c0 2024-04-23T18:15:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 Processor -- start 2024-04-23T18:15:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 -- start start 2024-04-23T18:15:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64148190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64152800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b640adc50 con 0x556b64152dd0 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b6404a830 con 0x556b64147db0 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68de612740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556b640a9c10 con 0x556b64148760 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41236/0 (socket says 172.21.15.32:41236) 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 -- 172.21.15.32:0/77423300 learned_addr learned my addr 172.21.15.32:0/77423300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d95bf640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 -- 172.21.15.32:0/77423300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 msgr2=0x556b64152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68ca7fc640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 0x7f68c00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 -- 172.21.15.32:0/77423300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 msgr2=0x556b64148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d8dbe640 1 -- 172.21.15.32:0/77423300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b63dc02d0 con 0x556b64152dd0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.394+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 0x7f68c00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68ca7fc640 1 -- 172.21.15.32:0/2303245232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68d00f1300 con 0x7f68d40694c0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f68c400dbe0 tx=0x7f68c400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2609bf6cfbc6c6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68bf7fe640 1 -- 172.21.15.32:0/77423300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c4005020 con 0x556b64152dd0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68bf7fe640 1 -- 172.21.15.32:0/77423300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68c40047a0 con 0x556b64152dd0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.395+0000 7f68bf7fe640 1 -- 172.21.15.32:0/77423300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40116a0 con 0x556b64152dd0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- 172.21.15.32:0/77423300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 msgr2=0x556b64155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 0x7f68c00a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f68cc005fd0 tx=0x7f68cc009440 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f68c400dbe0 tx=0x7f68c400f730 comp rx=0 tx=0).stop 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- 172.21.15.32:0/77423300 shutdown_connections 2024-04-23T18:15:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b64155220 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 --2- 172.21.15.32:0/77423300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- 172.21.15.32:0/77423300 >> 172.21.15.32:0/77423300 conn(0x556b64144070 msgr2=0x556b64144450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- 172.21.15.32:0/77423300 shutdown_connections 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- 172.21.15.32:0/77423300 wait complete. 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.396+0000 7f68de612740 1 -- start start 2024-04-23T18:15:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 Processor -- start 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64148760 0x556b641501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64152dd0 0x556b64150720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b64155d40 con 0x556b64147db0 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b64048470 con 0x556b64148760 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556b64155ec0 con 0x556b64152dd0 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68de612740 10 client.0 _ll_register_callbacks cb 0x556b6406df30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41252/0 (socket says 172.21.15.32:41252) 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 -- 172.21.15.32:0/2213394379 learned_addr learned my addr 172.21.15.32:0/2213394379 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d8dbe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64148760 0x556b641501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64152dd0 0x556b64150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 -- 172.21.15.32:0/2213394379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64152dd0 msgr2=0x556b64150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64152dd0 0x556b64150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 -- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64148760 msgr2=0x556b641501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64148760 0x556b641501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.397+0000 7f68d95bf640 1 -- 172.21.15.32:0/2213394379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b63e8a6b0 con 0x556b64147db0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.398+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005ec0 tx=0x7f68d0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44450bbf10f96ed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.398+0000 7f68bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0049070 con 0x556b64147db0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68d00bd580 con 0x556b64147db0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68c40047a0 con 0x556b64147db0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f68d00bcef0 con 0x556b64147db0 2024-04-23T18:15:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 10 client.15525 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556b64161fa0 con 0x556b64147db0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68c40047a0 con 0x556b64147db0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f68d00c1100 con 0x556b64147db0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 10 client.15525 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 10 client.15525 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.399+0000 7f68de612740 20 client.15525 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b64165a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68de612740 4 client.15525 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b640bb670 con 0x556b641645a0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b64165a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b64165a10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=609f6a2781689146 server_cookie=461f479cc6b11743 in_seq=0 out_seq=0 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.400+0000 7f68bcff9640 10 client.15525 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:15:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68bcff9640 1 -- 172.21.15.32:0/2213394379 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b640bb670 con 0x556b641645a0 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68bcff9640 10 client.15525 handle_command_reply: tid=0 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 client.15525 shutdown 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 msgr2=0x556b6414fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005ec0 tx=0x7f68d0009560 comp rx=0 tx=0).stop 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 shutdown_connections 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b64165a10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68bcff9640 0 client.15525 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64152dd0 0x556b64150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64148760 0x556b641501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 --2- 172.21.15.32:0/2213394379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64147db0 0x556b6414fca0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 >> 172.21.15.32:0/2213394379 conn(0x556b64144070 msgr2=0x556b64149ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 shutdown_connections 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 1 -- 172.21.15.32:0/2213394379 wait complete. 2024-04-23T18:15:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.401+0000 7f68de612740 20 client.15525 trim_cache size 0 max 16384 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 Processor -- start 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 -- start start 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64144450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b6414a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b64163d50 con 0x556b64152dd0 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b64164420 con 0x556b64147db0 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68de612740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556b64164af0 con 0x556b64148760 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41262/0 (socket says 172.21.15.32:41262) 2024-04-23T18:15:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2862418044 learned_addr learned my addr 172.21.15.32:0/2862418044 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b6414a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2862418044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 msgr2=0x556b6414a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b6414a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2862418044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 msgr2=0x556b64144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64144450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2862418044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b64143ea0 con 0x556b64152dd0 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.420+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b6414a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.421+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0149b0 tx=0x7f68cc009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28eaed098b350e84 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.421+0000 7f68beffd640 1 -- 172.21.15.32:0/2862418044 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc025070 con 0x556b64152dd0 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.421+0000 7f68beffd640 1 -- 172.21.15.32:0/2862418044 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68cc019070 con 0x556b64152dd0 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.421+0000 7f68beffd640 1 -- 172.21.15.32:0/2862418044 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc004460 con 0x556b64152dd0 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- 172.21.15.32:0/2862418044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 msgr2=0x556b6414a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0149b0 tx=0x7f68cc009ad0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- 172.21.15.32:0/2862418044 shutdown_connections 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6414a740 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b6414a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- 172.21.15.32:0/2862418044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- 172.21.15.32:0/2862418044 >> 172.21.15.32:0/2862418044 conn(0x556b64144070 msgr2=0x556b6414e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- 172.21.15.32:0/2862418044 shutdown_connections 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- 172.21.15.32:0/2862418044 wait complete. 2024-04-23T18:15:16.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- start start 2024-04-23T18:15:16.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 Processor -- start 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64167a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64167f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b64165890 con 0x556b64152dd0 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b641651c0 con 0x556b64147db0 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.422+0000 7f68de612740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556b64165f60 con 0x556b64148760 2024-04-23T18:15:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68de612740 10 client.0 _ll_register_callbacks cb 0x7f68d00ee810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41270/0 (socket says 172.21.15.32:41270) 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2275080612 learned_addr learned my addr 172.21.15.32:0/2275080612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d8dbe640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64167f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64167a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2275080612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 msgr2=0x556b64167f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64167f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 msgr2=0x556b64167a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64167a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 -- 172.21.15.32:0/2275080612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b64168870 con 0x556b64152dd0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64167a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68d9dc0640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f68cc017040 tx=0x7f68cc003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4700b7c7e46cdc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc025040 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68cc004b20 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68cc019070 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f68cc00f930 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68de612740 10 client.15531 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.423+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556b641cabb0 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68cc019070 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f68cc03e480 con 0x556b64152dd0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 10 client.15531 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 10 client.15531 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 20 client.15531 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b641cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 4 client.15531 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 --> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] -- command(tid 0: {"prefix": "status"}) -- 0x556b640b54e0 con 0x556b641645a0 2024-04-23T18:15:16.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b641cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68d95bf640 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b641cdfe0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d36c4b4087d32b80 server_cookie=cbebc87499bef0d6 in_seq=0 out_seq=0 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.424+0000 7f68bcff9640 10 client.15531 ms_handle_connect on v2:172.21.15.40:6836/1154751886 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68bcff9640 1 -- 172.21.15.32:0/2275080612 <== mds.1 v2:172.21.15.40:6836/1154751886 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556b640b54e0 con 0x556b641645a0 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68bcff9640 10 client.15531 handle_command_reply: tid=0 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 client.15531 shutdown 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 msgr2=0x556b6416c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f68cc017040 tx=0x7f68cc003b80 comp rx=0 tx=0).stop 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 shutdown_connections 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:6836/1154751886,v1:172.21.15.40:6837/1154751886] conn(0x556b641645a0 0x556b641cdfe0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b64152dd0 0x556b6416c200 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556b64148760 0x556b64167f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 --2- 172.21.15.32:0/2275080612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b64147db0 0x556b64167a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68bcff9640 0 client.15531 ms_handle_reset on v2:172.21.15.40:6836/1154751886 2024-04-23T18:15:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 >> 172.21.15.32:0/2275080612 conn(0x556b64144070 msgr2=0x556b6415e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 shutdown_connections 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 1 -- 172.21.15.32:0/2275080612 wait complete. 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.425+0000 7f68de612740 20 client.15531 trim_cache size 0 max 16384 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24515, 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:15:16.426 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 128.39649046100001, 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 22, 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 89, 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 147.63756021099999 2024-04-23T18:15:16.427 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 msgr2=0x7f68c00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 0x7f68c00a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f68cc005fd0 tx=0x7f68cc009440 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 msgr2=0x7f68d419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005940 tx=0x7f68d00042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 shutdown_connections 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68d4198670 0x7f68d4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d41073a0 0x7f68d419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f68c00a3db0 0x7f68c00a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 --2- 172.21.15.32:0/2303245232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d40694c0 0x7f68d419e470 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 >> 172.21.15.32:0/2303245232 conn(0x7f68d4076f20 msgr2=0x7f68d40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 shutdown_connections 2024-04-23T18:15:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.426+0000 7f68db823640 1 -- 172.21.15.32:0/2303245232 wait complete. 2024-04-23T18:15:16.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:15:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 -- 172.21.15.32:0/1063680409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90801073a0 msgr2=0x7f908010ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 --2- 172.21.15.32:0/1063680409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90801073a0 0x7f908010ef20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9078009ca0 tx=0x7f907802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 -- 172.21.15.32:0/1063680409 shutdown_connections 2024-04-23T18:15:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 --2- 172.21.15.32:0/1063680409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90801073a0 0x7f908010ef20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 --2- 172.21.15.32:0/1063680409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9080069e70 0x7f9080106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 --2- 172.21.15.32:0/1063680409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90800694c0 0x7f90800698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 -- 172.21.15.32:0/1063680409 >> 172.21.15.32:0/1063680409 conn(0x7f9080076f20 msgr2=0x7f9080077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 -- 172.21.15.32:0/1063680409 shutdown_connections 2024-04-23T18:15:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.542+0000 7f908733d640 1 -- 172.21.15.32:0/1063680409 wait complete. 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 Processor -- start 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 -- start start 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 0x7f908019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90801985c0 0x7f9080198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f90801122c0 con 0x7f90800694c0 2024-04-23T18:15:16.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9080112140 con 0x7f90801985c0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f908733d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9080112440 con 0x7f90801073a0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41290/0 (socket says 172.21.15.32:41290) 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 -- 172.21.15.32:0/4006672925 learned_addr learned my addr 172.21.15.32:0/4006672925 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90858da640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90801985c0 0x7f9080198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90848d8640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 0x7f908019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 -- 172.21.15.32:0/4006672925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 msgr2=0x7f908019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 0x7f908019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 -- 172.21.15.32:0/4006672925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90801985c0 msgr2=0x7f9080198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.543+0000 7f90850d9640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90801985c0 0x7f9080198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90850d9640 1 -- 172.21.15.32:0/4006672925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90801990e0 con 0x7f90800694c0 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90848d8640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 0x7f908019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90850d9640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f907000e9f0 tx=0x7f907000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90767fc640 1 -- 172.21.15.32:0/4006672925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907000cd80 con 0x7f90800694c0 2024-04-23T18:15:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90801993d0 con 0x7f90800694c0 2024-04-23T18:15:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90767fc640 1 -- 172.21.15.32:0/4006672925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90700108e0 con 0x7f90800694c0 2024-04-23T18:15:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f90767fc640 1 -- 172.21.15.32:0/4006672925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070018980 con 0x7f90800694c0 2024-04-23T18:15:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.544+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90801a5670 con 0x7f90800694c0 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.546+0000 7f90767fc640 1 -- 172.21.15.32:0/4006672925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f9070010430 con 0x7f90800694c0 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 Processor -- start 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 -- start start 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90767fc640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 0x7f906c0aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 0x55a7d3f00800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90858da640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 0x7f906c0aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f03220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90767fc640 1 -- 172.21.15.32:0/4006672925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f90700d0ce0 con 0x7f90800694c0 2024-04-23T18:15:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3e5bc50 con 0x55a7d3ef5db0 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3df8830 con 0x55a7d3f00dd0 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f908a04b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3e57c10 con 0x55a7d3ef6760 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41298/0 (socket says 172.21.15.32:41298) 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 -- 172.21.15.32:0/3520516385 learned_addr learned my addr 172.21.15.32:0/3520516385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90848d8640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 0x55a7d3f00800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90858da640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f03220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 -- 172.21.15.32:0/3520516385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 msgr2=0x55a7d3f00800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 0x55a7d3f00800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 -- 172.21.15.32:0/3520516385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 msgr2=0x55a7d3f03220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f03220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90850d9640 1 -- 172.21.15.32:0/3520516385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d3b6e2d0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.547+0000 7f90848d8640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 0x55a7d3f00800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.548+0000 7f90850d9640 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f90700ce450 tx=0x7f907009c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e58104e55ce4990 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.548+0000 7f90858da640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 0x7f906c0aa680 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9078006010 tx=0x7f9078031af0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.548+0000 7f906b7fe640 1 -- 172.21.15.32:0/3520516385 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700a0f10 con 0x55a7d3ef5db0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.548+0000 7f906b7fe640 1 -- 172.21.15.32:0/3520516385 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90700a57d0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.548+0000 7f906b7fe640 1 -- 172.21.15.32:0/3520516385 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700ce670 con 0x55a7d3ef5db0 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- 172.21.15.32:0/3520516385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 msgr2=0x55a7d3ef6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f90700ce450 tx=0x7f907009c5c0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- 172.21.15.32:0/3520516385 shutdown_connections 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f03220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef6760 0x55a7d3f00800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- 172.21.15.32:0/3520516385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3ef6190 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- 172.21.15.32:0/3520516385 >> 172.21.15.32:0/3520516385 conn(0x55a7d3ef2070 msgr2=0x55a7d3ef2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- 172.21.15.32:0/3520516385 shutdown_connections 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- 172.21.15.32:0/3520516385 wait complete. 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- start start 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 Processor -- start 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3efe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.549+0000 7f908a04b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f03ce0 con 0x55a7d3ef6760 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f908a04b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3df6470 con 0x55a7d3ef5db0 2024-04-23T18:15:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f908a04b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f03e60 con 0x55a7d3f00dd0 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f908a04b740 10 client.0 _ll_register_callbacks cb 0x55a7d3e1bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41310/0 (socket says 172.21.15.32:41310) 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 -- 172.21.15.32:0/1223330790 learned_addr learned my addr 172.21.15.32:0/1223330790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90858da640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3efe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90850d9640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 -- 172.21.15.32:0/1223330790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 msgr2=0x55a7d3efe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3efe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 -- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 msgr2=0x55a7d3efdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 -- 172.21.15.32:0/1223330790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d3c386b0 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90850d9640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f90848d8640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f908006b770 tx=0x7f907c0108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b100bcd90adc647a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f9068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c016070 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f907c012920 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90700a57d0 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.550+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f907c012b20 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 10 client.15543 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a7d3f0ffe0 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90700a57d0 con 0x55a7d3ef6760 2024-04-23T18:15:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f907c00f040 con 0x55a7d3ef6760 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 10 client.15543 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 10 client.15543 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 20 client.15543 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f12600 0x55a7d3f13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 4 client.15543 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7d3e69670 con 0x55a7d3f12600 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.551+0000 7f90850d9640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f12600 0x55a7d3f13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.552+0000 7f9068ff9640 10 client.15543 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.552+0000 7f90850d9640 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f12600 0x55a7d3f13a70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=100dc203ae4231a1 server_cookie=8ce7e5f065b4e519 in_seq=0 out_seq=0 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.552+0000 7f9068ff9640 1 -- 172.21.15.32:0/1223330790 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7d3e69670 con 0x55a7d3f12600 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.552+0000 7f9068ff9640 10 client.15543 handle_command_reply: tid=0 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.552+0000 7f908a04b740 1 client.15543 shutdown 2024-04-23T18:15:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 msgr2=0x55a7d3efe210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f908006b770 tx=0x7f907c0108d0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 shutdown_connections 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f12600 0x55a7d3f13a70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3efe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3efe210 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 --2- 172.21.15.32:0/1223330790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f9068ff9640 0 client.15543 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 >> 172.21.15.32:0/1223330790 conn(0x55a7d3ef2070 msgr2=0x55a7d3ef7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 shutdown_connections 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 1 -- 172.21.15.32:0/1223330790 wait complete. 2024-04-23T18:15:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.553+0000 7f908a04b740 20 client.15543 trim_cache size 0 max 16384 2024-04-23T18:15:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 Processor -- start 2024-04-23T18:15:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 -- start start 2024-04-23T18:15:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3ef8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f11db0 con 0x55a7d3ef6760 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f12480 con 0x55a7d3f00dd0 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f908a04b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f12b50 con 0x55a7d3ef5db0 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f90848d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f90848d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41322/0 (socket says 172.21.15.32:41322) 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.571+0000 7f90848d8640 1 -- 172.21.15.32:0/467816188 learned_addr learned my addr 172.21.15.32:0/467816188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90858da640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3ef8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90850d9640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 -- 172.21.15.32:0/467816188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 msgr2=0x55a7d3efe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 -- 172.21.15.32:0/467816188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 msgr2=0x55a7d3ef8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3ef8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 -- 172.21.15.32:0/467816188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d3ef98f0 con 0x55a7d3ef6760 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90850d9640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f90848d8640 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f907c000c00 tx=0x7f907c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95762862aafc7a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f906affd640 1 -- 172.21.15.32:0/467816188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c016070 con 0x55a7d3ef6760 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f906affd640 1 -- 172.21.15.32:0/467816188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f907c012920 con 0x55a7d3ef6760 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.572+0000 7f906affd640 1 -- 172.21.15.32:0/467816188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c02b070 con 0x55a7d3ef6760 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- 172.21.15.32:0/467816188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 msgr2=0x55a7d3ef8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f907c000c00 tx=0x7f907c009c30 comp rx=0 tx=0).stop 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- 172.21.15.32:0/467816188 shutdown_connections 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3f00dd0 0x55a7d3ef8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef6760 0x55a7d3ef8200 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 --2- 172.21.15.32:0/467816188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3ef5db0 0x55a7d3efe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- 172.21.15.32:0/467816188 >> 172.21.15.32:0/467816188 conn(0x55a7d3ef2070 msgr2=0x55a7d3efc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- 172.21.15.32:0/467816188 shutdown_connections 2024-04-23T18:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- 172.21.15.32:0/467816188 wait complete. 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 -- start start 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.573+0000 7f908a04b740 1 Processor -- start 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef6760 0x55a7d3f155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f15d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f138f0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f13220 con 0x55a7d3ef6760 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3f13fc0 con 0x55a7d3f00dd0 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f908a04b740 10 client.0 _ll_register_callbacks cb 0x7f907c009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41334/0 (socket says 172.21.15.32:41334) 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 -- 172.21.15.32:0/4188842079 learned_addr learned my addr 172.21.15.32:0/4188842079 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90848d8640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef6760 0x55a7d3f155d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90858da640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f15d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 -- 172.21.15.32:0/4188842079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 msgr2=0x55a7d3f15d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f15d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 -- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef6760 msgr2=0x55a7d3f155d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef6760 0x55a7d3f155d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 -- 172.21.15.32:0/4188842079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d3f19d40 con 0x55a7d3ef5db0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90858da640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f15d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f90850d9640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f907009c860 tx=0x7f9070009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f29c2da60e1229 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.574+0000 7f9068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700a57d0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f90700bd0b0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f907c012920 con 0x55a7d3ef5db0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f90700991d0 con 0x55a7d3ef5db0 2024-04-23T18:15:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 10 client.15549 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a7d3f78750 con 0x55a7d3ef5db0 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f907c012920 con 0x55a7d3ef5db0 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f90700cec40 con 0x55a7d3ef5db0 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 10 client.15549 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 10 client.15549 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 20 client.15549 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f7ce90 0x55a7d3f7bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 4 client.15549 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.575+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "status"}) -- 0x55a7d3e634e0 con 0x55a7d3f7ce90 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f90848d8640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f7ce90 0x55a7d3f7bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f9068ff9640 10 client.15549 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f90848d8640 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f7ce90 0x55a7d3f7bba0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=427eaee60c1f8b4a server_cookie=b1ea1ed53b76ede7 in_seq=0 out_seq=0 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f9068ff9640 1 -- 172.21.15.32:0/4188842079 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a7d3e634e0 con 0x55a7d3f7ce90 2024-04-23T18:15:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f9068ff9640 10 client.15549 handle_command_reply: tid=0 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f908a04b740 1 client.15549 shutdown 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 msgr2=0x55a7d3f1c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.576+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f907009c860 tx=0x7f9070009310 comp rx=0 tx=0).stop 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 shutdown_connections 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a7d3f7ce90 0x55a7d3f7bba0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a7d3f00dd0 0x55a7d3f15d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7d3ef6760 0x55a7d3f155d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f9068ff9640 0 client.15549 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 --2- 172.21.15.32:0/4188842079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7d3ef5db0 0x55a7d3f1c230 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 >> 172.21.15.32:0/4188842079 conn(0x55a7d3ef2070 msgr2=0x55a7d3f0c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 shutdown_connections 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 1 -- 172.21.15.32:0/4188842079 wait complete. 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908a04b740 20 client.15549 trim_cache size 0 max 16384 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cdb712f8-019b-11ef-bc93-c7b262605968", 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24551, 2024-04-23T18:15:16.579 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 130.57098185199999, 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 22, 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 89, 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 135.72466525199999 2024-04-23T18:15:16.580 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 msgr2=0x7f906c0aa680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 0x7f906c0aa680 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9078006010 tx=0x7f9078031af0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 msgr2=0x7f908019e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f907000e9f0 tx=0x7f907000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 shutdown_connections 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90801985c0 0x7f9080198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90801073a0 0x7f908019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f906c0a81c0 0x7f906c0aa680 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 --2- 172.21.15.32:0/4006672925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800694c0 0x7f908019e410 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.577+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 >> 172.21.15.32:0/4006672925 conn(0x7f9080076f20 msgr2=0x7f90800ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.578+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 shutdown_connections 2024-04-23T18:15:16.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:16.578+0000 7f908733d640 1 -- 172.21.15.32:0/4006672925 wait complete. 2024-04-23T18:15:16.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:16 smithi032 ceph-mon[29221]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:15:16.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1230534978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/20082469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:16 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1743914030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.583 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-23T18:15:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:16 smithi064 ceph-mon[33062]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:15:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1230534978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/20082469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:16 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1743914030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:16 smithi040 ceph-mon[32626]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:15:16.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1230534978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/20082469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:16.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:16 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1743914030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:17.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-23T18:15:17.645 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-23T18:15:17.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-23T18:15:17.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-23T18:15:17.801 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/: 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 data 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data: 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:17.802 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:15:17.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-23T18:15:17.885 INFO:teuthology.orchestra.run.smithi032.stdout:[Service] 2024-04-23T18:15:17.885 INFO:teuthology.orchestra.run.smithi032.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-23T18:15:17.885 INFO:teuthology.orchestra.run.smithi032.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-23T18:15:17.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-23T18:15:18.012 INFO:teuthology.orchestra.run.smithi032.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-23T18:15:18.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:18 smithi064 ceph-mon[33062]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:15:18.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:18 smithi032 ceph-mon[29221]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:15:18.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:18 smithi040 ceph-mon[32626]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:15:20.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:20 smithi064 ceph-mon[33062]: pgmap v9: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 94 KiB/s wr, 52 op/s 2024-04-23T18:15:20.719 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:20 smithi040 ceph-mon[32626]: pgmap v9: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 94 KiB/s wr, 52 op/s 2024-04-23T18:15:20.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:20 smithi032 ceph-mon[29221]: pgmap v9: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 94 KiB/s wr, 52 op/s 2024-04-23T18:15:22.259 INFO:teuthology.orchestra.run.smithi032.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-23T18:15:22.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-23T18:15:22.301 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/: 2024-04-23T18:15:22.301 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 data 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data: 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:total 35 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 . 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 base 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 23 18:15 global 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 log 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_commit_ts 2024-04-23T18:15:22.302 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_dynshmem 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 23 18:15 pg_hba.conf 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 23 18:15 pg_ident.conf 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 pg_logical 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 pg_multixact 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_notify 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_replslot 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_serial 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_snapshots 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_stat 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_stat_tmp 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 pg_subtrans 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_tblspc 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_twophase 2024-04-23T18:15:22.303 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 pg_wal 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 pg_xact 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 23 18:15 postgresql.auto.conf 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 23 18:15 postgresql.conf 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base: 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 . 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 1 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 13493 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 13494 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-23T18:15:22.304 INFO:teuthology.orchestra.run.smithi032.stdout:total 7777 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 . 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 .. 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 1247 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:15 1249 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:15 1259 2024-04-23T18:15:22.305 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:15:22.306 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:15:22.307 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2579 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:15:22.308 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:15:22.309 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:15 2608 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2610 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:15:22.310 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:15:22.311 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 23 18:15 2619 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2619_fsm 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:15 2658 2024-04-23T18:15:22.312 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 2659 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2662 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2663 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2664 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2665 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2666 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2667 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:15 2673 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:15 2674 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:15:22.313 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2678 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2679 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:15:22.314 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2696 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2703 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2704 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:15:22.315 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:15:22.316 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:15:22.317 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:15:22.318 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3455 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:15:22.319 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:15:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:15:22.321 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:15:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:15:22.323 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:15:22.326 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:15:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:total 7777 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 . 2024-04-23T18:15:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 .. 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 1247 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:15 1249 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:15 1259 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:15:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:15:22.330 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:15:22.331 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2579 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:15 2608 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2610 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 23 18:15 2619 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2619_fsm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:15 2658 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 2659 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2662 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2663 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2664 2024-04-23T18:15:22.336 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2665 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2666 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2667 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:15 2673 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:15 2674 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2678 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2679 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:15:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2696 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:15:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2703 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2704 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:15:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:15:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:15:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:15:22.342 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3455 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:15:22.343 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:15:22.344 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:15:22.345 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:15:22.346 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:15:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:15:22.348 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:total 7777 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 . 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 .. 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 1247 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:15:22.349 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:15 1249 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:15 1259 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:15:22.350 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2579 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:15:22.352 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:15:22.353 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:15 2608 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2610 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:15:22.354 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:15:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 23 18:15 2619 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2619_fsm 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:15 2658 2024-04-23T18:15:22.356 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 2659 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2662 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2663 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2664 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2665 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2666 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2667 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:15 2673 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:15 2674 2024-04-23T18:15:22.357 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2678 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2679 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2696 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2703 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2704 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:15:22.362 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3455 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:15:22.363 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:15:22.364 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:15:22.365 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:15:22.366 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:15:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:15:22.368 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/global: 2024-04-23T18:15:22.369 INFO:teuthology.orchestra.run.smithi032.stdout:total 537 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 23 18:15 . 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1213 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1213_fsm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1213_vm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1214 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1214_fsm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1214_vm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 1232 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 1233 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1260 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1260_fsm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1260_vm 2024-04-23T18:15:22.370 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1261 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1261_fsm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1261_vm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1262 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1262_fsm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1262_vm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2396 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2396_fsm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2396_vm 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2397 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2671 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2672 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2676 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2677 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2694 2024-04-23T18:15:22.371 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2695 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2697 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2698 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2846 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2847 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2964 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2965 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2966 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2967 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3592 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3593 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4060 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4061 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4175 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4176 2024-04-23T18:15:22.372 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4177 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4178 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4181 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4182 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4183 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4184 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4185 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4186 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6000 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6001 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6002 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6100 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6114 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6115 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 pg_control 2024-04-23T18:15:22.373 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/log: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.374 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:total 1 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 . 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 mappings 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 23 18:15 replorigin_checkpoint 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 snapshots 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 .. 2024-04-23T18:15:22.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 .. 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 . 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 members 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 offsets 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:15:22.376 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 .. 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 0000 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 .. 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 0000 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.378 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:22.379 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 0000 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-23T18:15:22.380 INFO:teuthology.orchestra.run.smithi032.stdout:total 16384 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 . 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:15 000000010000000000000001 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 archive_status 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 .. 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:15:22.381 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 23 18:15 .. 2024-04-23T18:15:22.382 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 0000 2024-04-23T18:15:22.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-23T18:15:22.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-23T18:15:22.529 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:22 smithi032 ceph-mon[29221]: pgmap v10: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 86 KiB/s wr, 48 op/s 2024-04-23T18:15:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:dropping old tables... 2024-04-23T18:15:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-23T18:15:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-23T18:15:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-23T18:15:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-23T18:15:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:creating tables... 2024-04-23T18:15:22.563 INFO:teuthology.orchestra.run.smithi032.stderr:generating data (client-side)... 2024-04-23T18:15:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.06 s, remaining 1.79 s) 2024-04-23T18:15:22.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:22 smithi064 ceph-mon[33062]: pgmap v10: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 86 KiB/s wr, 48 op/s 2024-04-23T18:15:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.22 s, remaining 3.29 s) 2024-04-23T18:15:22.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:22 smithi040 ceph-mon[32626]: pgmap v10: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 86 KiB/s wr, 48 op/s 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 -- 172.21.15.32:0/453665302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c107270 msgr2=0x7ff86c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 --2- 172.21.15.32:0/453665302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c107270 0x7ff86c10ee40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff864009ca0 tx=0x7ff86402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 -- 172.21.15.32:0/453665302 shutdown_connections 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 --2- 172.21.15.32:0/453665302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c107270 0x7ff86c10ee40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 --2- 172.21.15.32:0/453665302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c102b50 0x7ff86c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 --2- 172.21.15.32:0/453665302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c1021a0 0x7ff86c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 -- 172.21.15.32:0/453665302 >> 172.21.15.32:0/453665302 conn(0x7ff86c0fda20 msgr2=0x7ff86c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 -- 172.21.15.32:0/453665302 shutdown_connections 2024-04-23T18:15:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.881+0000 7ff873f29640 1 -- 172.21.15.32:0/453665302 wait complete. 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 Processor -- start 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 -- start start 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c1021a0 0x7ff86c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 0x7ff86c198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1121e0 con 0x7ff86c19d010 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c112060 con 0x7ff86c1021a0 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff873f29640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c112360 con 0x7ff86c107270 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40540/0 (socket says 172.21.15.32:40540) 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 -- 172.21.15.32:0/376252901 learned_addr learned my addr 172.21.15.32:0/376252901 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff871cc5640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c1021a0 0x7ff86c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8714c4640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 0x7ff86c198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 -- 172.21.15.32:0/376252901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 msgr2=0x7ff86c198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 0x7ff86c198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 -- 172.21.15.32:0/376252901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c1021a0 msgr2=0x7ff86c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c1021a0 0x7ff86c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8724c6640 1 -- 172.21.15.32:0/376252901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86c19db00 con 0x7ff86c19d010 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.882+0000 7ff8714c4640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 0x7ff86c198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff8724c6640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff864005cc0 tx=0x7ff864031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff862ffd640 1 -- 172.21.15.32:0/376252901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff864003b70 con 0x7ff86c19d010 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86c19dd90 con 0x7ff86c19d010 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff862ffd640 1 -- 172.21.15.32:0/376252901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8640318f0 con 0x7ff86c19d010 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff862ffd640 1 -- 172.21.15.32:0/376252901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86402f950 con 0x7ff86c19d010 2024-04-23T18:15:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.883+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff86c1a55e0 con 0x7ff86c19d010 2024-04-23T18:15:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff862ffd640 1 -- 172.21.15.32:0/376252901 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7ff86402fb50 con 0x7ff86c19d010 2024-04-23T18:15:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 Processor -- start 2024-04-23T18:15:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 -- start start 2024-04-23T18:15:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef4b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef4bc10 0x56453ef4e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56453eea1970 con 0x56453ef3c540 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56453ee3e4a0 con 0x56453ef4bc10 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff876c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56453ee9d930 con 0x56453ef3ce60 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40548/0 (socket says 172.21.15.32:40548) 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.886+0000 7ff8714c4640 1 -- 172.21.15.32:0/1508038422 learned_addr learned my addr 172.21.15.32:0/1508038422 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef4bc10 0x56453ef4e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff862ffd640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 0x7ff8580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff871cc5640 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef4b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff862ffd640 1 -- 172.21.15.32:0/376252901 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff8640f1bd0 con 0x7ff86c19d010 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 -- 172.21.15.32:0/1508038422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 msgr2=0x56453ef4b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef4b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 -- 172.21.15.32:0/1508038422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef4bc10 msgr2=0x56453ef4e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef4bc10 0x56453ef4e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8724c6640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 0x7ff8580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 -- 172.21.15.32:0/1508038422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56453ebb4210 con 0x56453ef3c540 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005f60 tx=0x7ff85c009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d041ab67ce09cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff857fff640 1 -- 172.21.15.32:0/1508038422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c013d70 con 0x56453ef3c540 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff8724c6640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 0x7ff8580a6260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8640ef420 tx=0x7ff8640095c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff857fff640 1 -- 172.21.15.32:0/1508038422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c0130d0 con 0x56453ef3c540 2024-04-23T18:15:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.887+0000 7ff857fff640 1 -- 172.21.15.32:0/1508038422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c0116d0 con 0x56453ef3c540 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 -- 172.21.15.32:0/1508038422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 msgr2=0x56453ef3c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005f60 tx=0x7ff85c009630 comp rx=0 tx=0).stop 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 -- 172.21.15.32:0/1508038422 shutdown_connections 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef4bc10 0x56453ef4e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef4b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 --2- 172.21.15.32:0/1508038422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef3c920 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 -- 172.21.15.32:0/1508038422 >> 172.21.15.32:0/1508038422 conn(0x56453ef38b30 msgr2=0x56453ef38f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 -- 172.21.15.32:0/1508038422 shutdown_connections 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.888+0000 7ff876c42740 1 -- 172.21.15.32:0/1508038422 wait complete. 2024-04-23T18:15:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 -- start start 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 Processor -- start 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3c540 0x56453ef4b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef48820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56453ef4e7e0 con 0x56453ef4bc10 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56453ee3c0b0 con 0x56453ef3c540 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56453ef4e960 con 0x56453ef3ce60 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff876c42740 10 client.0 _ll_register_callbacks cb 0x56453ee29d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff871cc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef48820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40552/0 (socket says 172.21.15.32:40552) 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff8714c4640 1 -- 172.21.15.32:0/1011429409 learned_addr learned my addr 172.21.15.32:0/1011429409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.889+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3c540 0x56453ef4b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 -- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3c540 msgr2=0x56453ef4b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3c540 0x56453ef4b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 -- 172.21.15.32:0/1011429409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 msgr2=0x56453ef48d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 -- 172.21.15.32:0/1011429409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56453ec80620 con 0x56453ef3ce60 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff871cc5640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef48820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8680126b0 tx=0x7ff868012b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=269965d527741976 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8557fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868004020 con 0x56453ef3ce60 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff8680041c0 con 0x56453ef3ce60 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff85c0130d0 con 0x56453ef3ce60 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff8680043c0 con 0x56453ef3ce60 2024-04-23T18:15:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff876c42740 10 client.25225 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56453ef5a760 con 0x56453ef3ce60 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.890+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c0130d0 con 0x56453ef3ce60 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff868003bc0 con 0x56453ef3ce60 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 10 client.25225 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 10 client.25225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 20 client.25225 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453ef5bc20 0x56453ef5e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 4 client.25225 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56453eeaf390 con 0x56453ef5bc20 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453ef5bc20 0x56453ef5e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff8557fa640 10 client.25225 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.891+0000 7ff8714c4640 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453ef5bc20 0x56453ef5e010 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b27e551a3e13ed71 server_cookie=17ac79a41a495e in_seq=0 out_seq=0 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff8557fa640 1 -- 172.21.15.32:0/1011429409 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56453eeaf390 con 0x56453ef5bc20 2024-04-23T18:15:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff8557fa640 10 client.25225 handle_command_reply: tid=0 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 client.25225 shutdown 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 msgr2=0x56453ef48820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef48820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8680126b0 tx=0x7ff868012b70 comp rx=0 tx=0).stop 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 shutdown_connections 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453ef5bc20 0x56453ef5e010 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef48d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3ce60 0x56453ef48820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 --2- 172.21.15.32:0/1011429409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3c540 0x56453ef4b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff8557fa640 0 client.25225 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 >> 172.21.15.32:0/1011429409 conn(0x56453ef38b30 msgr2=0x56453ef3e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 shutdown_connections 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 1 -- 172.21.15.32:0/1011429409 wait complete. 2024-04-23T18:15:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.892+0000 7ff876c42740 20 client.25225 trim_cache size 0 max 16384 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.908+0000 7ff876c42740 1 Processor -- start 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.908+0000 7ff876c42740 1 -- start start 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef43f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef37b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1ae120 con 0x56453ef4bc10 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1ab9f0 con 0x56453ef3ce60 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff876c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1a9840 con 0x56453ef3c540 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:45110/0 (socket says 172.21.15.32:45110) 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff871cc5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef43f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 -- 172.21.15.32:0/377398319 learned_addr learned my addr 172.21.15.32:0/377398319 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef37b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 -- 172.21.15.32:0/377398319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 msgr2=0x56453ef43f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef43f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 -- 172.21.15.32:0/377398319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 msgr2=0x56453ef37b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef37b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 -- 172.21.15.32:0/377398319 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56453ef38960 con 0x56453ef3c540 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8714c4640 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0133d0 tx=0x7ff85c0098d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291f85318445824a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8577fe640 1 -- 172.21.15.32:0/377398319 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c0096a0 con 0x56453ef3c540 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8577fe640 1 -- 172.21.15.32:0/377398319 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c005580 con 0x56453ef3c540 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.909+0000 7ff8577fe640 1 -- 172.21.15.32:0/377398319 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c031800 con 0x56453ef3c540 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 -- 172.21.15.32:0/377398319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 msgr2=0x56453ef576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0133d0 tx=0x7ff85c0098d0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 -- 172.21.15.32:0/377398319 shutdown_connections 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef4bc10 0x56453ef37b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef43f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 --2- 172.21.15.32:0/377398319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef3c540 0x56453ef576a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 -- 172.21.15.32:0/377398319 >> 172.21.15.32:0/377398319 conn(0x56453ef38b30 msgr2=0x56453ef37420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 -- 172.21.15.32:0/377398319 shutdown_connections 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.910+0000 7ff876c42740 1 -- 172.21.15.32:0/377398319 wait complete. 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 -- start start 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 Processor -- start 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef54030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef4bc10 0x56453ef4f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1a9eb0 con 0x56453ef3c540 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1ab320 con 0x56453ef3ce60 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1aa580 con 0x56453ef4bc10 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff876c42740 10 client.0 _ll_register_callbacks cb 0x7ff868009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff871cc5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef54030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff8714c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40580/0 (socket says 172.21.15.32:40580) 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.911+0000 7ff8714c4640 1 -- 172.21.15.32:0/4059231573 learned_addr learned my addr 172.21.15.32:0/4059231573 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 -- 172.21.15.32:0/4059231573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef4bc10 msgr2=0x56453ef4f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef4bc10 0x56453ef4f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 -- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 msgr2=0x56453ef54030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef54030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 -- 172.21.15.32:0/4059231573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56453ef4f830 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff871cc5640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef54030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8714c4640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009b80 tx=0x7ff85c0095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63d295459d7ac95 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8557fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c00fa40 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff85c00fbe0 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff85c005580 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff85c005080 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff876c42740 10 client.15567 fetch_fsmap learned FSMap version 22 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56453efc1d20 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.912+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c005580 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff85c003820 con 0x56453ef3c540 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 10 client.15567 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:15:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 20 client.15567 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453efc22e0 0x56453efc4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 4 client.15567 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56453eea9200 con 0x56453efc22e0 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff871cc5640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453efc22e0 0x56453efc4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff8557fa640 10 client.15567 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:22.913+0000 7ff871cc5640 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453efc22e0 0x56453efc4700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c3256e48904bc2f server_cookie=73b0ebdcf9d8458 in_seq=0 out_seq=0 2024-04-23T18:15:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.51 s, remaining 4.96 s) 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.100+0000 7ff8557fa640 1 -- 172.21.15.32:0/4059231573 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56453eea9200 con 0x56453efc22e0 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.100+0000 7ff8557fa640 10 client.15567 handle_command_reply: tid=0 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.100+0000 7ff876c42740 1 client.15567 shutdown 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 msgr2=0x56453ef53af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009b80 tx=0x7ff85c0095b0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 shutdown_connections 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56453efc22e0 0x56453efc4700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56453ef4bc10 0x56453ef4f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56453ef3ce60 0x56453ef54030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff8557fa640 0 client.15567 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 --2- 172.21.15.32:0/4059231573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56453ef3c540 0x56453ef53af0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 >> 172.21.15.32:0/4059231573 conn(0x56453ef38b30 msgr2=0x56453ef37cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 shutdown_connections 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 1 -- 172.21.15.32:0/4059231573 wait complete. 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff876c42740 20 client.15567 trim_cache size 0 max 16384 2024-04-23T18:15:23.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:15:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:15:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:15:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:15:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 msgr2=0x7ff8580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.101+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 0x7ff8580a6260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8640ef420 tx=0x7ff8640095c0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 msgr2=0x7ff86c19d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff864005cc0 tx=0x7ff864031700 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 shutdown_connections 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff8580a3da0 0x7ff8580a6260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff86c19d010 0x7ff86c19d4c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff86c107270 0x7ff86c198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 --2- 172.21.15.32:0/376252901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff86c1021a0 0x7ff86c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 >> 172.21.15.32:0/376252901 conn(0x7ff86c0fda20 msgr2=0x7ff86c0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 shutdown_connections 2024-04-23T18:15:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:23.102+0000 7ff873f29640 1 -- 172.21.15.32:0/376252901 wait complete. 2024-04-23T18:15:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.57 s, remaining 4.02 s) 2024-04-23T18:15:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.74 s, remaining 3.99 s) 2024-04-23T18:15:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.90 s, remaining 3.90 s) 2024-04-23T18:15:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.97 s, remaining 3.46 s) 2024-04-23T18:15:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.14 s, remaining 3.42 s) 2024-04-23T18:15:23.800 INFO:teuthology.orchestra.run.smithi032.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.22 s, remaining 3.12 s) 2024-04-23T18:15:23.978 INFO:teuthology.orchestra.run.smithi032.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.40 s, remaining 3.07 s) 2024-04-23T18:15:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.55 s, remaining 2.96 s) 2024-04-23T18:15:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.65 s, remaining 2.74 s) 2024-04-23T18:15:24.375 INFO:teuthology.orchestra.run.smithi032.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.79 s, remaining 2.62 s) 2024-04-23T18:15:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.91 s, remaining 2.46 s) 2024-04-23T18:15:24.641 INFO:teuthology.orchestra.run.smithi032.stderr:1500000 of 3200000 tuples (46%) done (elapsed 2.06 s, remaining 2.34 s) 2024-04-23T18:15:24.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:24 smithi064 ceph-mon[33062]: pgmap v11: 97 pgs: 97 active+clean; 21 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 2.7 MiB/s wr, 81 op/s 2024-04-23T18:15:24.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:24 smithi032 ceph-mon[29221]: pgmap v11: 97 pgs: 97 active+clean; 21 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 2.7 MiB/s wr, 81 op/s 2024-04-23T18:15:24.778 INFO:teuthology.orchestra.run.smithi032.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.20 s, remaining 2.20 s) 2024-04-23T18:15:24.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:24 smithi040 ceph-mon[32626]: pgmap v11: 97 pgs: 97 active+clean; 21 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 2.7 MiB/s wr, 81 op/s 2024-04-23T18:15:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.29 s, remaining 2.02 s) 2024-04-23T18:15:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.47 s, remaining 1.92 s) 2024-04-23T18:15:25.181 INFO:teuthology.orchestra.run.smithi032.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.60 s, remaining 1.78 s) 2024-04-23T18:15:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.70 s, remaining 1.62 s) 2024-04-23T18:15:25.427 INFO:teuthology.orchestra.run.smithi032.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.85 s, remaining 1.49 s) 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2200000 of 3200000 tuples (68%) done (elapsed 3.02 s, remaining 1.37 s) 2024-04-23T18:15:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2300000 of 3200000 tuples (71%) done (elapsed 3.12 s, remaining 1.22 s) 2024-04-23T18:15:25.828 INFO:teuthology.orchestra.run.smithi032.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.25 s, remaining 1.08 s) 2024-04-23T18:15:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.41 s, remaining 0.96 s) 2024-04-23T18:15:26.098 INFO:teuthology.orchestra.run.smithi032.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.52 s, remaining 0.81 s) 2024-04-23T18:15:26.230 INFO:teuthology.orchestra.run.smithi032.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.65 s, remaining 0.68 s) 2024-04-23T18:15:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.75 s, remaining 0.54 s) 2024-04-23T18:15:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.92 s, remaining 0.41 s) 2024-04-23T18:15:26.637 INFO:teuthology.orchestra.run.smithi032.stderr:3000000 of 3200000 tuples (93%) done (elapsed 4.06 s, remaining 0.27 s) 2024-04-23T18:15:26.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:26 smithi064 ceph-mon[33062]: pgmap v12: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 12 MiB/s wr, 330 op/s 2024-04-23T18:15:26.732 INFO:teuthology.orchestra.run.smithi032.stderr:3100000 of 3200000 tuples (96%) done (elapsed 4.15 s, remaining 0.13 s) 2024-04-23T18:15:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:26 smithi032 ceph-mon[29221]: pgmap v12: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 12 MiB/s wr, 330 op/s 2024-04-23T18:15:26.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:26 smithi040 ceph-mon[32626]: pgmap v12: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 12 MiB/s wr, 330 op/s 2024-04-23T18:15:26.890 INFO:teuthology.orchestra.run.smithi032.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.31 s, remaining 0.00 s) 2024-04-23T18:15:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:vacuuming... 2024-04-23T18:15:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:creating primary keys... 2024-04-23T18:15:27.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:27 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:27.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:27 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:27.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:27 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:28 smithi064 ceph-mon[33062]: pgmap v13: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 12 MiB/s wr, 286 op/s 2024-04-23T18:15:28.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:28 smithi032 ceph-mon[29221]: pgmap v13: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 12 MiB/s wr, 286 op/s 2024-04-23T18:15:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:done in 6.21 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 4.34 s, vacuum 0.62 s, primary keys 1.25 s). 2024-04-23T18:15:28.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-23T18:15:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:starting vacuum...end. 2024-04-23T18:15:28.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:28 smithi040 ceph-mon[32626]: pgmap v13: 97 pgs: 97 active+clean; 93 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 12 MiB/s wr, 286 op/s 2024-04-23T18:15:30.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:30 smithi064 ceph-mon[33062]: pgmap v14: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 65 MiB/s wr, 324 op/s 2024-04-23T18:15:30.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:30 smithi032 ceph-mon[29221]: pgmap v14: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 65 MiB/s wr, 324 op/s 2024-04-23T18:15:30.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:30 smithi040 ceph-mon[32626]: pgmap v14: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 65 MiB/s wr, 324 op/s 2024-04-23T18:15:31.584 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:15:31.584 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:15:31.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:31.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 -- 172.21.15.32:0/4045619287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 msgr2=0x7f4b28101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 --2- 172.21.15.32:0/4045619287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b28101500 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009900 tx=0x7f4b1c02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 -- 172.21.15.32:0/4045619287 shutdown_connections 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 --2- 172.21.15.32:0/4045619287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28101a40 0x7f4b2810dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 --2- 172.21.15.32:0/4045619287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b28101500 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 --2- 172.21.15.32:0/4045619287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28106ed0 0x7f4b281072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 -- 172.21.15.32:0/4045619287 >> 172.21.15.32:0/4045619287 conn(0x7f4b28077c40 msgr2=0x7f4b28078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 -- 172.21.15.32:0/4045619287 shutdown_connections 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.729+0000 7f4b2e433640 1 -- 172.21.15.32:0/4045619287 wait complete. 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 Processor -- start 2024-04-23T18:15:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 -- start start 2024-04-23T18:15:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28101a40 0x7f4b281a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 0x7f4b2819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28114450 con 0x7f4b281010a0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281142d0 con 0x7f4b28101a40 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2e433640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281145d0 con 0x7f4b28106ed0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b27fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b27fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60288/0 (socket says 172.21.15.32:60288) 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b27fff640 1 -- 172.21.15.32:0/1213885820 learned_addr learned my addr 172.21.15.32:0/1213885820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b2c9d0640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 0x7f4b2819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.730+0000 7f4b277fe640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28101a40 0x7f4b281a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 -- 172.21.15.32:0/1213885820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 msgr2=0x7f4b2819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 0x7f4b2819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 -- 172.21.15.32:0/1213885820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28101a40 msgr2=0x7f4b281a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28101a40 0x7f4b281a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 -- 172.21.15.32:0/1213885820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2819b050 con 0x7f4b281010a0 2024-04-23T18:15:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b2c9d0640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 0x7f4b2819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b27fff640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800d950 tx=0x7f4b1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18017070 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b18009c30 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.731+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800f620 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.732+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2819b340 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.732+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b281a7420 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.734+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b281025b0 con 0x7f4b281010a0 2024-04-23T18:15:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.734+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f4b18010460 con 0x7f4b281010a0 2024-04-23T18:15:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.735+0000 7f4b257fa640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 0x7f4b100a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.735+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4b180d0350 con 0x7f4b281010a0 2024-04-23T18:15:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.736+0000 7f4b277fe640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 0x7f4b100a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.737+0000 7f4b277fe640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 0x7f4b100a63b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c005ab0 tx=0x7f4b1c005a20 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:31.741+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b18009de0 con 0x7f4b281010a0 2024-04-23T18:15:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.063+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b28109310 con 0x7f4b281010a0 2024-04-23T18:15:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.064+0000 7f4b257fa640 1 -- 172.21.15.32:0/1213885820 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11854 (secure 0 0 0) 0x7f4b1809c280 con 0x7f4b281010a0 2024-04-23T18:15:32.058 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:15:32.059 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:30.858665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:15:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-04-23T18:15:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 msgr2=0x7f4b100a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 0x7f4b100a63b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c005ab0 tx=0x7f4b1c005a20 comp rx=0 tx=0).stop 2024-04-23T18:15:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 msgr2=0x7f4b281a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800d950 tx=0x7f4b1800de10 comp rx=0 tx=0).stop 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 shutdown_connections 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b28106ed0 0x7f4b2819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b28101a40 0x7f4b281a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b100a3ef0 0x7f4b100a63b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 --2- 172.21.15.32:0/1213885820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b281010a0 0x7f4b281a0680 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.067+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 >> 172.21.15.32:0/1213885820 conn(0x7f4b28077c40 msgr2=0x7f4b280ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.068+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 shutdown_connections 2024-04-23T18:15:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.068+0000 7f4b2e433640 1 -- 172.21.15.32:0/1213885820 wait complete. 2024-04-23T18:15:32.076 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:15:32.076 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:15:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.238+0000 7f75e37fe640 1 -- 172.21.15.32:0/981190777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40307c0 con 0x7f75ec106190 2024-04-23T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 -- 172.21.15.32:0/981190777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec106190 msgr2=0x7f75ec10dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 --2- 172.21.15.32:0/981190777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec106190 0x7f75ec10dd60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009ca0 tx=0x7f75e402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 -- 172.21.15.32:0/981190777 shutdown_connections 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 --2- 172.21.15.32:0/981190777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec106190 0x7f75ec10dd60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 --2- 172.21.15.32:0/981190777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec101a70 0x7f75ec105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 --2- 172.21.15.32:0/981190777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec1010c0 0x7f75ec1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 -- 172.21.15.32:0/981190777 >> 172.21.15.32:0/981190777 conn(0x7f75ec078030 msgr2=0x7f75ec078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 -- 172.21.15.32:0/981190777 shutdown_connections 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.239+0000 7f75f2b51640 1 -- 172.21.15.32:0/981190777 wait complete. 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 Processor -- start 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 -- start start 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 0x7f75ec06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 0x7f75ec19c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec111130 con 0x7f75ec1010c0 2024-04-23T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec110fb0 con 0x7f75ec06e770 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f2b51640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1112b0 con 0x7f75ec106190 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f08ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f08ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60294/0 (socket says 172.21.15.32:60294) 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f08ed640 1 -- 172.21.15.32:0/884448419 learned_addr learned my addr 172.21.15.32:0/884448419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75e3fff640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 0x7f75ec06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.240+0000 7f75f10ee640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 0x7f75ec19c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 -- 172.21.15.32:0/884448419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 msgr2=0x7f75ec06e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 0x7f75ec06e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 -- 172.21.15.32:0/884448419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 msgr2=0x7f75ec19c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 0x7f75ec19c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 -- 172.21.15.32:0/884448419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec19c6b0 con 0x7f75ec1010c0 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f10ee640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 0x7f75ec19c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:32.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75e3fff640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 0x7f75ec06e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f08ed640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f75e800e9f0 tx=0x7f75e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75e1ffb640 1 -- 172.21.15.32:0/884448419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e800cd80 con 0x7f75ec1010c0 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75e1ffb640 1 -- 172.21.15.32:0/884448419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e80108e0 con 0x7f75ec1010c0 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75e1ffb640 1 -- 172.21.15.32:0/884448419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8018980 con 0x7f75ec1010c0 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec19c8e0 con 0x7f75ec1010c0 2024-04-23T18:15:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.241+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec19cdb0 con 0x7f75ec1010c0 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.244+0000 7f75e1ffb640 1 -- 172.21.15.32:0/884448419 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f75e8010430 con 0x7f75ec1010c0 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.244+0000 7f75f585f740 1 Processor -- start 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.244+0000 7f75f585f740 1 -- start start 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.244+0000 7f75f585f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.244+0000 7f75f585f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed3704b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f585f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed3707480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f585f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ed365f370 con 0x560ed36fa0a0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f585f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ed35fc2e0 con 0x560ed36fa9c0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f585f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560ed365b330 con 0x560ed3705090 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60310/0 (socket says 172.21.15.32:60310) 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 -- 172.21.15.32:0/2477374268 learned_addr learned my addr 172.21.15.32:0/2477374268 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75e3fff640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed3707480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f08ed640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed3704b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 -- 172.21.15.32:0/2477374268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 msgr2=0x560ed3707480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed3707480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 -- 172.21.15.32:0/2477374268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 msgr2=0x560ed3704b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed3704b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 -- 172.21.15.32:0/2477374268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ed343dd80 con 0x560ed36fa0a0 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f08ed640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed3704b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75e1ffb640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 0x7f75d40b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009400 tx=0x7f75e4009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219ad6857288b736 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75f08ed640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 0x7f75d40b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75d2ffd640 1 -- 172.21.15.32:0/2477374268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40307c0 con 0x560ed36fa0a0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75d2ffd640 1 -- 172.21.15.32:0/2477374268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e4036070 con 0x560ed36fa0a0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.245+0000 7f75e1ffb640 1 -- 172.21.15.32:0/884448419 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f75e80d0ce0 con 0x7f75ec1010c0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.246+0000 7f75d2ffd640 1 -- 172.21.15.32:0/2477374268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4030bc0 con 0x560ed36fa0a0 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.246+0000 7f75f585f740 1 -- 172.21.15.32:0/2477374268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 msgr2=0x560ed36fa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.246+0000 7f75f585f740 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009400 tx=0x7f75e4009800 comp rx=0 tx=0).stop 2024-04-23T18:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.246+0000 7f75f08ed640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 0x7f75d40b7280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75e80ce420 tx=0x7f75e8018420 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.247+0000 7f75f585f740 1 -- 172.21.15.32:0/2477374268 shutdown_connections 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.247+0000 7f75f585f740 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed3707480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.247+0000 7f75f585f740 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed3704b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.247+0000 7f75f585f740 1 --2- 172.21.15.32:0/2477374268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed36fa480 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.247+0000 7f75f585f740 1 -- 172.21.15.32:0/2477374268 >> 172.21.15.32:0/2477374268 conn(0x560ed36f63c0 msgr2=0x560ed36f67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.248+0000 7f75f585f740 1 -- 172.21.15.32:0/2477374268 shutdown_connections 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.248+0000 7f75f585f740 1 -- 172.21.15.32:0/2477374268 wait complete. 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.248+0000 7f75f585f740 1 -- start start 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.248+0000 7f75f585f740 1 Processor -- start 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa0a0 0x560ed3701f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed37029f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ed3707e70 con 0x560ed36fa9c0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ed35f9fb0 con 0x560ed36fa0a0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560ed3707ff0 con 0x560ed3705090 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f585f740 10 client.0 _ll_register_callbacks cb 0x560ed35f99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed37029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60314/0 (socket says 172.21.15.32:60314) 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 -- 172.21.15.32:0/888567265 learned_addr learned my addr 172.21.15.32:0/888567265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75f10ee640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa0a0 0x560ed3701f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 -- 172.21.15.32:0/888567265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 msgr2=0x560ed37029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed37029f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 -- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa0a0 msgr2=0x560ed3701f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa0a0 0x560ed3701f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.249+0000 7f75e3fff640 1 -- 172.21.15.32:0/888567265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ed349e510 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75e3fff640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f75dc012d20 tx=0x7f75dc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3771fb96dab1bad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc012110 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75e4036070 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f75dc004050 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f75dc0122b0 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75f585f740 10 client.15585 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.250+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560ed3331670 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e4036070 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f75dc025430 con 0x560ed36fa9c0 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 10 client.15585 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 10 client.15585 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 20 client.15585 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3717780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 4 client.15585 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:32.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.251+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ed366cd90 con 0x560ed3715390 2024-04-23T18:15:32.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.252+0000 7f75f10ee640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3717780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.252+0000 7f75f10ee640 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3717780 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b62457db419c7d server_cookie=9e3fd24a671156e7 in_seq=0 out_seq=0 2024-04-23T18:15:32.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.252+0000 7f75cbfff640 10 client.15585 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75cbfff640 1 -- 172.21.15.32:0/888567265 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ed366cd90 con 0x560ed3715390 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75cbfff640 10 client.15585 handle_command_reply: tid=0 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 client.15585 shutdown 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 msgr2=0x560ed37024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f75dc012d20 tx=0x7f75dc00f790 comp rx=0 tx=0).stop 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 shutdown_connections 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3717780 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed37029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75cbfff640 0 client.15585 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa9c0 0x560ed37024b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 --2- 172.21.15.32:0/888567265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa0a0 0x560ed3701f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 >> 172.21.15.32:0/888567265 conn(0x560ed36f63c0 msgr2=0x560ed36fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 shutdown_connections 2024-04-23T18:15:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 1 -- 172.21.15.32:0/888567265 wait complete. 2024-04-23T18:15:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.253+0000 7f75f585f740 20 client.15585 trim_cache size 0 max 16384 2024-04-23T18:15:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 Processor -- start 2024-04-23T18:15:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 -- start start 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed36fc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed371d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a5a20 con 0x560ed36fa0a0 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a32f0 con 0x560ed36fa9c0 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f585f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a1160 con 0x560ed3705090 2024-04-23T18:15:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60318/0 (socket says 172.21.15.32:60318) 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f10ee640 1 -- 172.21.15.32:0/3783515242 learned_addr learned my addr 172.21.15.32:0/3783515242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75e3fff640 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed36fc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.279+0000 7f75f10ee640 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed371d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 -- 172.21.15.32:0/3783515242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 msgr2=0x560ed371d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed371d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 -- 172.21.15.32:0/3783515242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 msgr2=0x560ed36fc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed36fc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 -- 172.21.15.32:0/3783515242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ed3700f20 con 0x560ed36fa0a0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75f10ee640 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f75e4031040 tx=0x7f75e4005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8da265cf0b847a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75d27fc640 1 -- 172.21.15.32:0/3783515242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e405c430 con 0x560ed36fa0a0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75d27fc640 1 -- 172.21.15.32:0/3783515242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e4036040 con 0x560ed36fa0a0 2024-04-23T18:15:32.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.280+0000 7f75d27fc640 1 -- 172.21.15.32:0/3783515242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4030cf0 con 0x560ed36fa0a0 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- 172.21.15.32:0/3783515242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 msgr2=0x560ed3710d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f75e4031040 tx=0x7f75e4005f80 comp rx=0 tx=0).stop 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- 172.21.15.32:0/3783515242 shutdown_connections 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed3705090 0x560ed371d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed36fa9c0 0x560ed36fc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 --2- 172.21.15.32:0/3783515242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed3710d80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- 172.21.15.32:0/3783515242 >> 172.21.15.32:0/3783515242 conn(0x560ed36f63c0 msgr2=0x560ed37037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- 172.21.15.32:0/3783515242 shutdown_connections 2024-04-23T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- 172.21.15.32:0/3783515242 wait complete. 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 -- start start 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.281+0000 7f75f585f740 1 Processor -- start 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed36fa9c0 0x560ed3709960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed3705090 0x560ed3709ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a17b0 con 0x560ed36fa0a0 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a2c20 con 0x560ed3705090 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1a1e80 con 0x560ed36fa9c0 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f585f740 10 client.0 _ll_register_callbacks cb 0x7f75dc012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60332/0 (socket says 172.21.15.32:60332) 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 -- 172.21.15.32:0/2945368233 learned_addr learned my addr 172.21.15.32:0/2945368233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75e3fff640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed36fa9c0 0x560ed3709960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed3705090 0x560ed3709ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 -- 172.21.15.32:0/2945368233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed36fa9c0 msgr2=0x560ed3709960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed36fa9c0 0x560ed3709960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 -- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed3705090 msgr2=0x560ed3709ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed3705090 0x560ed3709ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 -- 172.21.15.32:0/2945368233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ed36fa7f0 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75f10ee640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f75e4033040 tx=0x7f75e40319c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8609c772bfe593a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.282+0000 7f75cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.283+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4062060 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.283+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f75e40306c0 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.283+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75e4036040 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f75e40308c0 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 10 client.15591 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560ed3784e80 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e4036040 con 0x560ed36fa0a0 2024-04-23T18:15:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f75e40449a0 con 0x560ed36fa0a0 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 10 client.15591 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 10 client.15591 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 20 client.15591 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3787060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 4 client.15591 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.284+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x560ed3666c00 con 0x560ed3715390 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.285+0000 7f75e3fff640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3787060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.285+0000 7f75e3fff640 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3787060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f7acada0f56e55 server_cookie=d2ba9ec3693357a8 in_seq=0 out_seq=0 2024-04-23T18:15:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:32.285+0000 7f75cbfff640 10 client.15591 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:32.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:32 smithi032 ceph-mon[29221]: pgmap v15: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 65 MiB/s wr, 320 op/s 2024-04-23T18:15:32.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:32 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:32.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1213885820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:32.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:32 smithi040 ceph-mon[32626]: pgmap v15: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 65 MiB/s wr, 320 op/s 2024-04-23T18:15:32.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:32 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:32.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1213885820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:32.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:32 smithi064 ceph-mon[33062]: pgmap v15: 97 pgs: 97 active+clean; 413 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 65 MiB/s wr, 320 op/s 2024-04-23T18:15:32.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:32 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:32.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1213885820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.395+0000 7f75cbfff640 1 -- 172.21.15.32:0/2945368233 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x560ed3666c00 con 0x560ed3715390 2024-04-23T18:15:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.395+0000 7f75cbfff640 10 client.15591 handle_command_reply: tid=0 2024-04-23T18:15:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.395+0000 7f75f585f740 1 client.15591 shutdown 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 msgr2=0x560ed370d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f75e4033040 tx=0x7f75e40319c0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 shutdown_connections 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560ed3715390 0x560ed3787060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ed3705090 0x560ed3709ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ed36fa9c0 0x560ed3709960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 --2- 172.21.15.32:0/2945368233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ed36fa0a0 0x560ed370d590 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75cbfff640 0 client.15591 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 >> 172.21.15.32:0/2945368233 conn(0x560ed36f63c0 msgr2=0x560ed3704030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 shutdown_connections 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 1 -- 172.21.15.32:0/2945368233 wait complete. 2024-04-23T18:15:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.396+0000 7f75f585f740 20 client.15591 trim_cache size 0 max 16384 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 24, 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 2, 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "ef875bc0": { 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 2, 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:15:33.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 msgr2=0x7f75d40b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:33.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 0x7f75d40b7280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75e80ce420 tx=0x7f75e8018420 comp rx=0 tx=0).stop 2024-04-23T18:15:33.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 msgr2=0x7f75ec06dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f75e800e9f0 tx=0x7f75e800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 shutdown_connections 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ec06e770 0x7f75ec19c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75ec106190 0x7f75ec06e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f75d40b4dc0 0x7f75d40b7280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 --2- 172.21.15.32:0/884448419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75ec1010c0 0x7f75ec06dcf0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 >> 172.21.15.32:0/884448419 conn(0x7f75ec078030 msgr2=0x7f75ec100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 shutdown_connections 2024-04-23T18:15:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:15:33.397+0000 7f75f2b51640 1 -- 172.21.15.32:0/884448419 wait complete. 2024-04-23T18:15:33.405 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ef875bc0', quiesce took 1.8 seconds. Will release after: 30.0 2024-04-23T18:15:33.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 5.0 s, 2579.1 tps, lat 25.849 ms stddev 25.594 2024-04-23T18:15:34.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:34 smithi032 ceph-mon[29221]: pgmap v16: 97 pgs: 97 active+clean; 452 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 67 MiB/s wr, 368 op/s 2024-04-23T18:15:34.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:34 smithi040 ceph-mon[32626]: pgmap v16: 97 pgs: 97 active+clean; 452 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 67 MiB/s wr, 368 op/s 2024-04-23T18:15:34.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:34 smithi064 ceph-mon[33062]: pgmap v16: 97 pgs: 97 active+clean; 452 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 67 MiB/s wr, 368 op/s 2024-04-23T18:15:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 -- 172.21.15.32:0/630101231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a810c5b0 msgr2=0x7f38a810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 --2- 172.21.15.32:0/630101231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a810c5b0 0x7f38a810e9e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009ca0 tx=0x7f38a002a950 comp rx=0 tx=0).stop 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 -- 172.21.15.32:0/630101231 shutdown_connections 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 --2- 172.21.15.32:0/630101231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a810c5b0 0x7f38a810e9e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 --2- 172.21.15.32:0/630101231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a8103b00 0x7f38a810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 --2- 172.21.15.32:0/630101231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a81031e0 0x7f38a81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 -- 172.21.15.32:0/630101231 >> 172.21.15.32:0/630101231 conn(0x7f38a8077fa0 msgr2=0x7f38a80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 -- 172.21.15.32:0/630101231 shutdown_connections 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.202+0000 7f38ae02a640 1 -- 172.21.15.32:0/630101231 wait complete. 2024-04-23T18:15:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 Processor -- start 2024-04-23T18:15:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 -- start start 2024-04-23T18:15:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a81031e0 0x7f38a8100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a810c5b0 0x7f38a81018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81121d0 con 0x7f38a8103b00 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38a8112050 con 0x7f38a81031e0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.203+0000 7f38ae02a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38a8112350 con 0x7f38a810c5b0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60352/0 (socket says 172.21.15.32:60352) 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 -- 172.21.15.32:0/234481247 learned_addr learned my addr 172.21.15.32:0/234481247 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a77fe640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a81031e0 0x7f38a8100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a7fff640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a810c5b0 0x7f38a81018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 -- 172.21.15.32:0/234481247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a810c5b0 msgr2=0x7f38a81018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a810c5b0 0x7f38a81018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 -- 172.21.15.32:0/234481247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a81031e0 msgr2=0x7f38a8100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a81031e0 0x7f38a8100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 -- 172.21.15.32:0/234481247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a81a6bd0 con 0x7f38a8103b00 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f389c00d950 tx=0x7f389c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a4ff9640 1 -- 172.21.15.32:0/234481247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c017070 con 0x7f38a8103b00 2024-04-23T18:15:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a4ff9640 1 -- 172.21.15.32:0/234481247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c009c30 con 0x7f38a8103b00 2024-04-23T18:15:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a81a6ec0 con 0x7f38a8103b00 2024-04-23T18:15:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38a4ff9640 1 -- 172.21.15.32:0/234481247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c00f620 con 0x7f38a8103b00 2024-04-23T18:15:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.204+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a81a73d0 con 0x7f38a8103b00 2024-04-23T18:15:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38a4ff9640 1 -- 172.21.15.32:0/234481247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f389c010460 con 0x7f38a8103b00 2024-04-23T18:15:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 Processor -- start 2024-04-23T18:15:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 -- start start 2024-04-23T18:15:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d662a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56330d5c7970 con 0x56330d66d800 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56330d5644a0 con 0x56330d662f40 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.207+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56330d5c3930 con 0x56330d662620 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a4ff9640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 0x7f38900a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d662a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60356/0 (socket says 172.21.15.32:60356) 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 -- 172.21.15.32:0/3171038911 learned_addr learned my addr 172.21.15.32:0/3171038911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a7fff640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 0x7f38900a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a77fe640 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a4ff9640 1 -- 172.21.15.32:0/234481247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f389c0d0410 con 0x7f38a8103b00 2024-04-23T18:15:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 -- 172.21.15.32:0/3171038911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 msgr2=0x56330d662a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d662a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 -- 172.21.15.32:0/3171038911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 msgr2=0x56330d66d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 -- 172.21.15.32:0/3171038911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56330d2da210 con 0x56330d66d800 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f389c000c00 tx=0x7f389c0986b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a085d24ce32b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f38a7fff640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 0x7f38900a62e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009430 tx=0x7f38a0005c50 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f388dffb640 1 -- 172.21.15.32:0/3171038911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c020070 con 0x56330d66d800 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f388dffb640 1 -- 172.21.15.32:0/3171038911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c09c2e0 con 0x56330d66d800 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.208+0000 7f388dffb640 1 -- 172.21.15.32:0/3171038911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c098030 con 0x56330d66d800 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 -- 172.21.15.32:0/3171038911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 msgr2=0x56330d66fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f389c000c00 tx=0x7f389c0986b0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 -- 172.21.15.32:0/3171038911 shutdown_connections 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66fbf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/3171038911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d662a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 -- 172.21.15.32:0/3171038911 >> 172.21.15.32:0/3171038911 conn(0x56330d65eb30 msgr2=0x56330d65ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 -- 172.21.15.32:0/3171038911 shutdown_connections 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.209+0000 7f38b0e1e740 1 -- 172.21.15.32:0/3171038911 wait complete. 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 -- start start 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 Processor -- start 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d66d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56330d670370 con 0x56330d66d800 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56330d5620b0 con 0x56330d662f40 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56330d6704f0 con 0x56330d662620 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38b0e1e740 10 client.0 _ll_register_callbacks cb 0x56330d54fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60366/0 (socket says 172.21.15.32:60366) 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38a7fff640 1 -- 172.21.15.32:0/1562259467 learned_addr learned my addr 172.21.15.32:0/1562259467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.210+0000 7f38a77fe640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d66d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 -- 172.21.15.32:0/1562259467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 msgr2=0x56330d66d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d66d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 -- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 msgr2=0x56330d66a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 -- 172.21.15.32:0/1562259467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56330d3a6620 con 0x56330d66d800 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a77fe640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d66d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38a7fff640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f38a0035000 tx=0x7f38a00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2c76ddc3b064fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f3886ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a004f030 con 0x56330d66d800 2024-04-23T18:15:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f389c09c2e0 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f38a0003e00 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f38a0058030 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38b0e1e740 10 client.15603 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.211+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56330d67c370 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c09c2e0 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f38a004b550 con 0x56330d66d800 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 10 client.15603 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 20 client.15603 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d67d830 0x56330d67fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 4 client.15603 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56330d5d5390 con 0x56330d67d830 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.212+0000 7f38a77fe640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d67d830 0x56330d67fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f38a77fe640 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d67d830 0x56330d67fc40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=658b7734b8520314 server_cookie=136d286a793c83dd in_seq=0 out_seq=0 2024-04-23T18:15:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f3886ffd640 10 client.15603 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f3886ffd640 1 -- 172.21.15.32:0/1562259467 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56330d5d5390 con 0x56330d67d830 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f3886ffd640 10 client.15603 handle_command_reply: tid=0 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f38b0e1e740 1 client.15603 shutdown 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 msgr2=0x56330d66aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.213+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f38a0035000 tx=0x7f38a00095c0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 shutdown_connections 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d67d830 0x56330d67fc40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d66aa40 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d66a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f3886ffd640 0 client.15603 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/1562259467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d66d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 >> 172.21.15.32:0/1562259467 conn(0x56330d65eb30 msgr2=0x56330d664860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 shutdown_connections 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 1 -- 172.21.15.32:0/1562259467 wait complete. 2024-04-23T18:15:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.214+0000 7f38b0e1e740 20 client.15603 trim_cache size 0 max 16384 2024-04-23T18:15:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 Processor -- start 2024-04-23T18:15:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 -- start start 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 0x56330d679230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 0x56330d685550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81b2490 con 0x56330d662f40 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81afd60 con 0x56330d662620 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81adbb0 con 0x56330d66d800 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60372/0 (socket says 172.21.15.32:60372) 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38a6ffd640 1 -- 172.21.15.32:0/57822231 learned_addr learned my addr 172.21.15.32:0/57822231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.229+0000 7f38a77fe640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 0x56330d679230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 -- 172.21.15.32:0/57822231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 msgr2=0x56330d685550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a7fff640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 0x56330d685550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 0x56330d685550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 -- 172.21.15.32:0/57822231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 msgr2=0x56330d679230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 0x56330d679230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 -- 172.21.15.32:0/57822231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56330d6802c0 con 0x56330d662f40 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a7fff640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 0x56330d685550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a77fe640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 0x56330d679230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f389c098cb0 tx=0x7f389c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab8da489122f1fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f388d7fa640 1 -- 172.21.15.32:0/57822231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0b9070 con 0x56330d662f40 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f388d7fa640 1 -- 172.21.15.32:0/57822231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c0b5040 con 0x56330d662f40 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.230+0000 7f388d7fa640 1 -- 172.21.15.32:0/57822231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c020040 con 0x56330d662f40 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- 172.21.15.32:0/57822231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 msgr2=0x56330d66cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f389c098cb0 tx=0x7f389c009f20 comp rx=0 tx=0).stop 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- 172.21.15.32:0/57822231 shutdown_connections 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d66d800 0x56330d685550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d662f40 0x56330d66cce0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/57822231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662620 0x56330d679230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- 172.21.15.32:0/57822231 >> 172.21.15.32:0/57822231 conn(0x56330d65eb30 msgr2=0x56330d665af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- 172.21.15.32:0/57822231 shutdown_connections 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- 172.21.15.32:0/57822231 wait complete. 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- start start 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 Processor -- start 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d6711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d671730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81ae220 con 0x56330d66d800 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81af690 con 0x56330d662f40 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.231+0000 7f38b0e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81ae8f0 con 0x56330d662620 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38b0e1e740 10 client.0 _ll_register_callbacks cb 0x7f38a002f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60382/0 (socket says 172.21.15.32:60382) 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 -- 172.21.15.32:0/2306987715 learned_addr learned my addr 172.21.15.32:0/2306987715 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a6ffd640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d671730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a77fe640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d6711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 -- 172.21.15.32:0/2306987715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 msgr2=0x56330d6711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d6711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 -- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 msgr2=0x56330d671730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d671730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 -- 172.21.15.32:0/2306987715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56330d6e2e70 con 0x56330d66d800 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38a7fff640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f38a002ae70 tx=0x7f38a002faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83121312a2fe4cd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f3886ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a004f040 con 0x56330d66d800 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f38a0059070 con 0x56330d66d800 2024-04-23T18:15:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.232+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f389c0b5040 con 0x56330d66d800 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f38a0034040 con 0x56330d66d800 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 10 client.15609 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56330d6e40b0 con 0x56330d66d800 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c0b5040 con 0x56330d66d800 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f38a00046e0 con 0x56330d66d800 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 10 client.15609 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 20 client.15609 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d6e4850 0x56330d6e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 4 client.15609 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56330d5cf200 con 0x56330d6e4850 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.233+0000 7f38a77fe640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d6e4850 0x56330d6e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.234+0000 7f38a77fe640 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d6e4850 0x56330d6e6c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c5c5189906aa5a4 server_cookie=21d9e54dc95396ec in_seq=0 out_seq=0 2024-04-23T18:15:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.234+0000 7f3886ffd640 10 client.15609 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f3886ffd640 1 -- 172.21.15.32:0/2306987715 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56330d5cf200 con 0x56330d6e4850 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f3886ffd640 10 client.15609 handle_command_reply: tid=0 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 client.15609 shutdown 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 msgr2=0x56330d671c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f38a002ae70 tx=0x7f38a002faf0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 shutdown_connections 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56330d6e4850 0x56330d6e6c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56330d66d800 0x56330d671c70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56330d662f40 0x56330d671730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f3886ffd640 0 client.15609 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 --2- 172.21.15.32:0/2306987715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56330d662620 0x56330d6711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 >> 172.21.15.32:0/2306987715 conn(0x56330d65eb30 msgr2=0x56330d65dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 shutdown_connections 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 1 -- 172.21.15.32:0/2306987715 wait complete. 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.295+0000 7f38b0e1e740 20 client.15609 trim_cache size 0 max 16384 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:15:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:15:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:15:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 msgr2=0x7f38900a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 0x7f38900a62e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009430 tx=0x7f38a0005c50 comp rx=0 tx=0).stop 2024-04-23T18:15:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 msgr2=0x7f38a8101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f389c00d950 tx=0x7f389c00de10 comp rx=0 tx=0).stop 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 shutdown_connections 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38a810c5b0 0x7f38a81018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f38900a3e20 0x7f38900a62e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8103b00 0x7f38a8101370 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 --2- 172.21.15.32:0/234481247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a81031e0 0x7f38a8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 >> 172.21.15.32:0/234481247 conn(0x7f38a8077fa0 msgr2=0x7f38a80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 shutdown_connections 2024-04-23T18:15:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:36.296+0000 7f38ae02a640 1 -- 172.21.15.32:0/234481247 wait complete. 2024-04-23T18:15:36.724 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:36 smithi064 ceph-mon[33062]: pgmap v17: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 75 MiB/s wr, 496 op/s 2024-04-23T18:15:36.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:36 smithi032 ceph-mon[29221]: pgmap v17: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 75 MiB/s wr, 496 op/s 2024-04-23T18:15:36.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:36 smithi040 ceph-mon[32626]: pgmap v17: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 75 MiB/s wr, 496 op/s 2024-04-23T18:15:38.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:38 smithi032 ceph-mon[29221]: pgmap v18: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 66 MiB/s wr, 247 op/s 2024-04-23T18:15:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 10.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:15:38.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:38 smithi040 ceph-mon[32626]: pgmap v18: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 66 MiB/s wr, 247 op/s 2024-04-23T18:15:38.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:38 smithi064 ceph-mon[33062]: pgmap v18: 97 pgs: 97 active+clean; 810 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 66 MiB/s wr, 247 op/s 2024-04-23T18:15:40.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:40 smithi032 ceph-mon[29221]: pgmap v19: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 66 MiB/s wr, 267 op/s 2024-04-23T18:15:40.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:40 smithi040 ceph-mon[32626]: pgmap v19: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 66 MiB/s wr, 267 op/s 2024-04-23T18:15:40.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:40 smithi064 ceph-mon[33062]: pgmap v19: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 66 MiB/s wr, 267 op/s 2024-04-23T18:15:42.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:42 smithi032 ceph-mon[29221]: pgmap v20: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:42.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:42 smithi040 ceph-mon[32626]: pgmap v20: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:42.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:42 smithi064 ceph-mon[33062]: pgmap v20: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:43.866 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 15.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:15:44.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:44 smithi032 ceph-mon[29221]: pgmap v21: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:44.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:44 smithi040 ceph-mon[32626]: pgmap v21: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:44.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:44 smithi064 ceph-mon[33062]: pgmap v21: 97 pgs: 97 active+clean; 807 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 13 MiB/s wr, 229 op/s 2024-04-23T18:15:46.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:46 smithi032 ceph-mon[29221]: pgmap v22: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 184 op/s 2024-04-23T18:15:46.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:46 smithi040 ceph-mon[32626]: pgmap v22: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 184 op/s 2024-04-23T18:15:46.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:46 smithi064 ceph-mon[33062]: pgmap v22: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 184 op/s 2024-04-23T18:15:48.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:48 smithi032 ceph-mon[29221]: pgmap v23: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-23T18:15:48.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:15:48.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:48 smithi040 ceph-mon[32626]: pgmap v23: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-23T18:15:48.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:48 smithi064 ceph-mon[33062]: pgmap v23: 97 pgs: 97 active+clean; 807 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 -- 172.21.15.32:0/4219310444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 msgr2=0x7f758410eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 --2- 172.21.15.32:0/4219310444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f758410eeb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 -- 172.21.15.32:0/4219310444 shutdown_connections 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 --2- 172.21.15.32:0/4219310444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f758410eeb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 --2- 172.21.15.32:0/4219310444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75841014a0 0x7f7584106c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 --2- 172.21.15.32:0/4219310444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 -- 172.21.15.32:0/4219310444 >> 172.21.15.32:0/4219310444 conn(0x7f7584076200 msgr2=0x7f7584076610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 -- 172.21.15.32:0/4219310444 shutdown_connections 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.396+0000 7f758b720640 1 -- 172.21.15.32:0/4219310444 wait complete. 2024-04-23T18:15:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 Processor -- start 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 -- start start 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 0x7f758419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75841122b0 con 0x7f7584107370 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7584112130 con 0x7f7584100af0 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f758b720640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7584112430 con 0x7f758419cfd0 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f7588cbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f7588cbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37152/0 (socket says 172.21.15.32:37152) 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f7588cbb640 1 -- 172.21.15.32:0/456711231 learned_addr learned my addr 172.21.15.32:0/456711231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.397+0000 7f75894bc640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7589cbd640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 0x7f758419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 -- 172.21.15.32:0/456711231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 msgr2=0x7f758419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 0x7f758419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 -- 172.21.15.32:0/456711231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 msgr2=0x7f7584198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 -- 172.21.15.32:0/456711231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758419dac0 con 0x7f7584107370 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7589cbd640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 0x7f758419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f75894bc640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f7588cbb640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f757800d720 tx=0x7f757800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f75827fc640 1 -- 172.21.15.32:0/456711231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578017070 con 0x7f7584107370 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758419ddb0 con 0x7f7584107370 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f75827fc640 1 -- 172.21.15.32:0/456711231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757800de30 con 0x7f7584107370 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f75827fc640 1 -- 172.21.15.32:0/456711231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578004be0 con 0x7f7584107370 2024-04-23T18:15:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.398+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75841a5550 con 0x7f7584107370 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f75827fc640 1 -- 172.21.15.32:0/456711231 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 23) ==== 138822+0+0 (secure 0 0 0) 0x7f7578009c30 con 0x7f7584107370 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 Processor -- start 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 -- start start 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560b6b539970 con 0x560b6b5df800 2024-04-23T18:15:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560b6b4d64a0 con 0x560b6b5d4f40 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f758e43a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560b6b535930 con 0x560b6b5d4620 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.401+0000 7f75827fc640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 0x7f756c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7589cbd640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 0x7f756c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37158/0 (socket says 172.21.15.32:37158) 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 -- 172.21.15.32:0/1879675420 learned_addr learned my addr 172.21.15.32:0/1879675420 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7589cbd640 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f75827fc640 1 -- 172.21.15.32:0/456711231 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f75780d02f0 con 0x7f7584107370 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f75894bc640 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 -- 172.21.15.32:0/1879675420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 msgr2=0x560b6b5d4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5d4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 -- 172.21.15.32:0/1879675420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 msgr2=0x560b6b5df2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5df2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 -- 172.21.15.32:0/1879675420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b6b24c210 con 0x560b6b5df800 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7588cbb640 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f757809be70 tx=0x7f757806ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac695161bb25b98 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f7589cbd640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 0x7f756c0a6260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f757c031440 tx=0x7f757c0040c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f756b7fe640 1 -- 172.21.15.32:0/1879675420 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578020070 con 0x560b6b5df800 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f756b7fe640 1 -- 172.21.15.32:0/1879675420 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757809c170 con 0x560b6b5df800 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.402+0000 7f756b7fe640 1 -- 172.21.15.32:0/1879675420 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75780cdc80 con 0x560b6b5df800 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 -- 172.21.15.32:0/1879675420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 msgr2=0x560b6b5e1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f757809be70 tx=0x7f757806ec60 comp rx=0 tx=0).stop 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 -- 172.21.15.32:0/1879675420 shutdown_connections 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5e1bf0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5df2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 --2- 172.21.15.32:0/1879675420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 -- 172.21.15.32:0/1879675420 >> 172.21.15.32:0/1879675420 conn(0x560b6b5d0b30 msgr2=0x560b6b5d0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 -- 172.21.15.32:0/1879675420 shutdown_connections 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.403+0000 7f758e43a740 1 -- 172.21.15.32:0/1879675420 wait complete. 2024-04-23T18:15:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 -- start start 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 Processor -- start 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5df170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5dc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560b6b5e2430 con 0x560b6b5df800 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560b6b4d40b0 con 0x560b6b5d4620 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560b6b5e25b0 con 0x560b6b5d4f40 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f758e43a740 10 client.0 _ll_register_callbacks cb 0x560b6b4c1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37170/0 (socket says 172.21.15.32:37170) 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f7589cbd640 1 -- 172.21.15.32:0/3806865521 learned_addr learned my addr 172.21.15.32:0/3806865521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.404+0000 7f75894bc640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5df170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7588cbb640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 -- 172.21.15.32:0/3806865521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 msgr2=0x560b6b5dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 -- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 msgr2=0x560b6b5df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5df170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 -- 172.21.15.32:0/3806865521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b6b318620 con 0x560b6b5df800 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f75894bc640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5df170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7588cbb640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32834c65ca4a06f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c04e070 con 0x560b6b5df800 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f757c005250 con 0x560b6b5df800 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f757809c170 con 0x560b6b5df800 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f757c05a050 con 0x560b6b5df800 2024-04-23T18:15:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f758e43a740 10 client.15621 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560b6b5ee3b0 con 0x560b6b5df800 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.405+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757809c170 con 0x560b6b5df800 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f757c03a040 con 0x560b6b5df800 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 10 client.15621 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 20 client.15621 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b5ef8e0 0x560b6b5f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 4 client.15621 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b6b547390 con 0x560b6b5ef8e0 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f75894bc640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b5ef8e0 0x560b6b5f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f7568ff9640 10 client.15621 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.406+0000 7f75894bc640 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b5ef8e0 0x560b6b5f1cd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0a0f46bebc3e9d server_cookie=4c759cbf4936613c in_seq=0 out_seq=0 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f7568ff9640 1 -- 172.21.15.32:0/3806865521 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b6b547390 con 0x560b6b5ef8e0 2024-04-23T18:15:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f7568ff9640 10 client.15621 handle_command_reply: tid=0 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 client.15621 shutdown 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 msgr2=0x560b6b5dc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c005a20 comp rx=0 tx=0).stop 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 shutdown_connections 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b5ef8e0 0x560b6b5f1cd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5dc9c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f7568ff9640 0 client.15621 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 --2- 172.21.15.32:0/3806865521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5df170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.407+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 >> 172.21.15.32:0/3806865521 conn(0x560b6b5d0b30 msgr2=0x560b6b5d68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.408+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 shutdown_connections 2024-04-23T18:15:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.408+0000 7f758e43a740 1 -- 172.21.15.32:0/3806865521 wait complete. 2024-04-23T18:15:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.408+0000 7f758e43a740 20 client.15621 trim_cache size 0 max 16384 2024-04-23T18:15:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 Processor -- start 2024-04-23T18:15:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 -- start start 2024-04-23T18:15:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75841b0330 con 0x560b6b5df800 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75841adc00 con 0x560b6b5d4620 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f758e43a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75841aba70 con 0x560b6b5d4f40 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37174/0 (socket says 172.21.15.32:37174) 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 -- 172.21.15.32:0/3571031061 learned_addr learned my addr 172.21.15.32:0/3571031061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 -- 172.21.15.32:0/3571031061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 msgr2=0x560b6b5f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 -- 172.21.15.32:0/3571031061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 msgr2=0x560b6b5eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f75894bc640 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.423+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5eb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f7589cbd640 1 -- 172.21.15.32:0/3571031061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b6b5f2330 con 0x560b6b5df800 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f75894bc640 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f7589cbd640 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f757c031520 tx=0x7f757c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3664c86a4bb12961 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f756affd640 1 -- 172.21.15.32:0/3571031061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c04e040 con 0x560b6b5df800 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f756affd640 1 -- 172.21.15.32:0/3571031061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757c031710 con 0x560b6b5df800 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f756affd640 1 -- 172.21.15.32:0/3571031061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c035040 con 0x560b6b5df800 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 -- 172.21.15.32:0/3571031061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 msgr2=0x560b6b5f7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f757c031520 tx=0x7f757c005dd0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 -- 172.21.15.32:0/3571031061 shutdown_connections 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b5f7b00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4f40 0x560b6b5f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 --2- 172.21.15.32:0/3571031061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4620 0x560b6b5eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.424+0000 7f758e43a740 1 -- 172.21.15.32:0/3571031061 >> 172.21.15.32:0/3571031061 conn(0x560b6b5d0b30 msgr2=0x560b6b5cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- 172.21.15.32:0/3571031061 shutdown_connections 2024-04-23T18:15:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- 172.21.15.32:0/3571031061 wait complete. 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- start start 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 Processor -- start 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5e5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5e5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75841ac0c0 con 0x560b6b5df800 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75841ad530 con 0x560b6b5d4f40 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75841ac790 con 0x560b6b5d4620 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f758e43a740 10 client.0 _ll_register_callbacks cb 0x7f757c0043a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f7589cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37178/0 (socket says 172.21.15.32:37178) 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.425+0000 7f7589cbd640 1 -- 172.21.15.32:0/2846900173 learned_addr learned my addr 172.21.15.32:0/2846900173 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f75894bc640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5e5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7588cbb640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5e5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 -- 172.21.15.32:0/2846900173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 msgr2=0x560b6b5e5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5e5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 -- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 msgr2=0x560b6b5e5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5e5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 -- 172.21.15.32:0/2846900173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b6b5d59f0 con 0x560b6b5df800 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f75894bc640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7589cbd640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d6147f34625170 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c062020 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f757c0046c0 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f757c031710 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f757c06d050 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f758e43a740 10 client.15627 fetch_fsmap learned FSMap version 24 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.426+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560b6b656230 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757c031710 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f757c04a720 con 0x560b6b5df800 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 10 client.15627 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 20 client.15627 populate_metadata read hostname 'smithi032' 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b656940 0x560b6b658d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 4 client.15627 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:15:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b6b541200 con 0x560b6b656940 2024-04-23T18:15:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f75894bc640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b656940 0x560b6b658d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f7568ff9640 10 client.15627 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.427+0000 7f75894bc640 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b656940 0x560b6b658d60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44513c5d5160568e server_cookie=3cc0788e0d54864f in_seq=0 out_seq=0 2024-04-23T18:15:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f7568ff9640 1 -- 172.21.15.32:0/2846900173 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b6b541200 con 0x560b6b656940 2024-04-23T18:15:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f7568ff9640 10 client.15627 handle_command_reply: tid=0 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 client.15627 shutdown 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 msgr2=0x560b6b654e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c003cf0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 shutdown_connections 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560b6b656940 0x560b6b658d60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b6b5df800 0x560b6b654e40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b6b5d4f40 0x560b6b5e5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f7568ff9640 0 client.15627 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 --2- 172.21.15.32:0/2846900173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560b6b5d4620 0x560b6b5e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 >> 172.21.15.32:0/2846900173 conn(0x560b6b5d0b30 msgr2=0x560b6b5ea2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 shutdown_connections 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 1 -- 172.21.15.32:0/2846900173 wait complete. 2024-04-23T18:15:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.442+0000 7f758e43a740 20 client.15627 trim_cache size 0 max 16384 2024-04-23T18:15:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:15:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:15:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:15:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 msgr2=0x7f756c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 0x7f756c0a6260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f757c031440 tx=0x7f757c0040c0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 msgr2=0x7f7584198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f757800d720 tx=0x7f757800dbe0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 shutdown_connections 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f758419cfd0 0x7f758419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f756c0a3da0 0x7f756c0a6260 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7584107370 0x7f7584198c50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 --2- 172.21.15.32:0/456711231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7584100af0 0x7f7584198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 >> 172.21.15.32:0/456711231 conn(0x7f7584076200 msgr2=0x7f75840fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 shutdown_connections 2024-04-23T18:15:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:15:49.443+0000 7f758b720640 1 -- 172.21.15.32:0/456711231 wait complete. 2024-04-23T18:15:50.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:50 smithi032 ceph-mon[29221]: pgmap v24: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:15:50.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:50 smithi040 ceph-mon[32626]: pgmap v24: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:15:50.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:50 smithi064 ceph-mon[33062]: pgmap v24: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:15:52.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:52 smithi032 ceph-mon[29221]: pgmap v25: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-23T18:15:52.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:52 smithi040 ceph-mon[32626]: pgmap v25: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-23T18:15:52.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:52 smithi064 ceph-mon[33062]: pgmap v25: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-23T18:15:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:15:54.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:54 smithi032 ceph-mon[29221]: pgmap v26: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-04-23T18:15:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:54 smithi040 ceph-mon[32626]: pgmap v26: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-04-23T18:15:54.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:54 smithi064 ceph-mon[33062]: pgmap v26: 97 pgs: 97 active+clean; 807 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-04-23T18:15:56.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:56 smithi032 ceph-mon[29221]: pgmap v27: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-23T18:15:56.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:56 smithi040 ceph-mon[32626]: pgmap v27: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-23T18:15:56.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:56 smithi064 ceph-mon[33062]: pgmap v27: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-23T18:15:58.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:15:58 smithi032 ceph-mon[29221]: pgmap v28: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-04-23T18:15:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:15:58.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:15:58 smithi040 ceph-mon[32626]: pgmap v28: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-04-23T18:15:58.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:15:58 smithi064 ceph-mon[33062]: pgmap v28: 97 pgs: 97 active+clean; 807 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-04-23T18:16:00.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-mon[29221]: pgmap v29: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 33 op/s 2024-04-23T18:16:00.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:00 smithi040 ceph-mon[32626]: pgmap v29: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 33 op/s 2024-04-23T18:16:00.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:00 smithi064 ceph-mon[33062]: pgmap v29: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 33 op/s 2024-04-23T18:16:01.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:16:00.868+0000 7f452c93d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:01.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:16:00.868+0000 7f452c93d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:01.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:16:00.868+0000 7f452c93d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:01.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:16:00.868+0000 7f452c93d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:01.258 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:16:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:16:00.868+0000 7f452c93d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 -- 172.21.15.32:0/2301659076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1381014a0 msgr2=0x7fa138105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 --2- 172.21.15.32:0/2301659076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1381014a0 0x7fa138105290 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009900 tx=0x7fa12c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 -- 172.21.15.32:0/2301659076 shutdown_connections 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 --2- 172.21.15.32:0/2301659076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1381059c0 0x7fa13810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 --2- 172.21.15.32:0/2301659076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1381014a0 0x7fa138105290 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 --2- 172.21.15.32:0/2301659076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa138100af0 0x7fa138100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 -- 172.21.15.32:0/2301659076 >> 172.21.15.32:0/2301659076 conn(0x7fa138076200 msgr2=0x7fa138076610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 -- 172.21.15.32:0/2301659076 shutdown_connections 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.535+0000 7fa13dda2640 1 -- 172.21.15.32:0/2301659076 wait complete. 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 Processor -- start 2024-04-23T18:16:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 -- start start 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa138100af0 0x7fa138198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1381059c0 0x7fa138198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa138112240 con 0x7fa13819cfb0 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381120c0 con 0x7fa1381059c0 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa13dda2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381123c0 con 0x7fa138100af0 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59622/0 (socket says 172.21.15.32:59622) 2024-04-23T18:16:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 -- 172.21.15.32:0/939065629 learned_addr learned my addr 172.21.15.32:0/939065629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa1377fe640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa138100af0 0x7fa138198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa136ffd640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1381059c0 0x7fa138198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 -- 172.21.15.32:0/939065629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa138100af0 msgr2=0x7fa138198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa138100af0 0x7fa138198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 -- 172.21.15.32:0/939065629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1381059c0 msgr2=0x7fa138198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1381059c0 0x7fa138198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.536+0000 7fa137fff640 1 -- 172.21.15.32:0/939065629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13819daa0 con 0x7fa13819cfb0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa137fff640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa13000eaa0 tx=0x7fa13000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa134ff9640 1 -- 172.21.15.32:0/939065629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13000cdb0 con 0x7fa13819cfb0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa134ff9640 1 -- 172.21.15.32:0/939065629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa130005020 con 0x7fa13819cfb0 2024-04-23T18:16:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13819dd90 con 0x7fa13819cfb0 2024-04-23T18:16:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa134ff9640 1 -- 172.21.15.32:0/939065629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300116d0 con 0x7fa13819cfb0 2024-04-23T18:16:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.537+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1381a5550 con 0x7fa13819cfb0 2024-04-23T18:16:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.539+0000 7fa134ff9640 1 -- 172.21.15.32:0/939065629 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fa1300051d0 con 0x7fa13819cfb0 2024-04-23T18:16:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 Processor -- start 2024-04-23T18:16:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 -- start start 2024-04-23T18:16:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa134ff9640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 0x7fa1200a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x565172899900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899e40 0x5651728a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5651727fe970 con 0x5651728a8c10 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56517279b4a0 con 0x565172899520 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa140b96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5651727fa930 con 0x565172899e40 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa136ffd640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 0x7fa1200a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa134ff9640 1 -- 172.21.15.32:0/939065629 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa1300d0af0 con 0x7fa13819cfb0 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59628/0 (socket says 172.21.15.32:59628) 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 -- 172.21.15.32:0/656753850 learned_addr learned my addr 172.21.15.32:0/656753850 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa1377fe640 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x565172899900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa136ffd640 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899e40 0x5651728a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 -- 172.21.15.32:0/656753850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899e40 msgr2=0x5651728a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899e40 0x5651728a86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 -- 172.21.15.32:0/656753850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 msgr2=0x565172899900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x565172899900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa137fff640 1 -- 172.21.15.32:0/656753850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565172511210 con 0x5651728a8c10 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.540+0000 7fa136ffd640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 0x7fa1200a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c032c60 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa137fff640 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa13009c8d0 tx=0x7fa130007280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff87a773c7ea9bc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa11dffb640 1 -- 172.21.15.32:0/656753850 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13001f040 con 0x5651728a8c10 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa11dffb640 1 -- 172.21.15.32:0/656753850 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1300a0c40 con 0x5651728a8c10 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa11dffb640 1 -- 172.21.15.32:0/656753850 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300054f0 con 0x5651728a8c10 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 -- 172.21.15.32:0/656753850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 msgr2=0x5651728ab000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa13009c8d0 tx=0x7fa130007280 comp rx=0 tx=0).stop 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 -- 172.21.15.32:0/656753850 shutdown_connections 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651728a8c10 0x5651728ab000 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899e40 0x5651728a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 --2- 172.21.15.32:0/656753850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x565172899900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 -- 172.21.15.32:0/656753850 >> 172.21.15.32:0/656753850 conn(0x565172895b30 msgr2=0x565172895f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 -- 172.21.15.32:0/656753850 shutdown_connections 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.541+0000 7fa140b96740 1 -- 172.21.15.32:0/656753850 wait complete. 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 -- start start 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 Processor -- start 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x5651728a8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5651728ab780 con 0x565172899e40 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5651727990b0 con 0x565172899520 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5651728ab900 con 0x5651728a8c10 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa140b96740 10 client.0 _ll_register_callbacks cb 0x565172786d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59642/0 (socket says 172.21.15.32:59642) 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa136ffd640 1 -- 172.21.15.32:0/2421723217 learned_addr learned my addr 172.21.15.32:0/2421723217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.542+0000 7fa1377fe640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x5651728a8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa137fff640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 -- 172.21.15.32:0/2421723217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 msgr2=0x5651728a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 -- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 msgr2=0x5651728a8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x5651728a8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 -- 172.21.15.32:0/2421723217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651725dd620 con 0x565172899e40 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa137fff640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa136ffd640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa12c033000 tx=0x7fa12c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfebcb50e336ae97 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa116ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c04d020 con 0x565172899e40 2024-04-23T18:16:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa12c005040 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1300a0c40 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa12c058050 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa140b96740 10 client.15645 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5651728b7730 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.543+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1300a0c40 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa12c048070 con 0x565172899e40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 10 client.15645 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 10 client.15645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 20 client.15645 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5651728b8c40 0x5651728bb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 4 client.15645 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56517280c390 con 0x5651728b8c40 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa1377fe640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5651728b8c40 0x5651728bb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa116ffd640 10 client.15645 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.544+0000 7fa1377fe640 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5651728b8c40 0x5651728bb050 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc5382822bf0562f server_cookie=cae916a96dfcaa1f in_seq=0 out_seq=0 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa116ffd640 1 -- 172.21.15.32:0/2421723217 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56517280c390 con 0x5651728b8c40 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa116ffd640 10 client.15645 handle_command_reply: tid=0 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 client.15645 shutdown 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 msgr2=0x5651728a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa12c033000 tx=0x7fa12c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 shutdown_connections 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5651728b8c40 0x5651728bb050 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa116ffd640 0 client.15645 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a5860 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 --2- 172.21.15.32:0/2421723217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899520 0x5651728a8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 >> 172.21.15.32:0/2421723217 conn(0x565172895b30 msgr2=0x56517289b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 shutdown_connections 2024-04-23T18:16:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 1 -- 172.21.15.32:0/2421723217 wait complete. 2024-04-23T18:16:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.545+0000 7fa140b96740 20 client.15645 trim_cache size 0 max 16384 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 Processor -- start 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 -- start start 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899520 0x5651728b4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 0x5651728a80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381ae1a0 con 0x565172899e40 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381aba70 con 0x5651728a8c10 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa140b96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381a9900 con 0x565172899520 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59652/0 (socket says 172.21.15.32:59652) 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.560+0000 7fa136ffd640 1 -- 172.21.15.32:0/771462270 learned_addr learned my addr 172.21.15.32:0/771462270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa1377fe640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899520 0x5651728b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 -- 172.21.15.32:0/771462270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899520 msgr2=0x5651728b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899520 0x5651728b4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 -- 172.21.15.32:0/771462270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 msgr2=0x5651728a80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa137fff640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 0x5651728a80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 0x5651728a80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 -- 172.21.15.32:0/771462270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651728bb6d0 con 0x565172899e40 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa137fff640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 0x5651728a80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa136ffd640 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa12c02a930 tx=0x7fa12c0314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160adf46ef6637c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa11d7fa640 1 -- 172.21.15.32:0/771462270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c04d030 con 0x565172899e40 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa11d7fa640 1 -- 172.21.15.32:0/771462270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12c0043d0 con 0x565172899e40 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa11d7fa640 1 -- 172.21.15.32:0/771462270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c035040 con 0x565172899e40 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa140b96740 1 -- 172.21.15.32:0/771462270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 msgr2=0x5651728a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.561+0000 7fa140b96740 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa12c02a930 tx=0x7fa12c0314e0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- 172.21.15.32:0/771462270 shutdown_connections 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651728a8c10 0x5651728a80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899e40 0x5651728a0d80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- 172.21.15.32:0/771462270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565172899520 0x5651728b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- 172.21.15.32:0/771462270 >> 172.21.15.32:0/771462270 conn(0x565172895b30 msgr2=0x5651728a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- 172.21.15.32:0/771462270 shutdown_connections 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- 172.21.15.32:0/771462270 wait complete. 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- start start 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 Processor -- start 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 0x5651728ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381a9f50 con 0x565172899520 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381ab3a0 con 0x565172899e40 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381aa600 con 0x5651728a8c10 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa140b96740 10 client.0 _ll_register_callbacks cb 0x7fa12c031570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.562+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59660/0 (socket says 172.21.15.32:59660) 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 -- 172.21.15.32:0/3378960461 learned_addr learned my addr 172.21.15.32:0/3378960461 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa137fff640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa136ffd640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 0x5651728ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 -- 172.21.15.32:0/3378960461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 msgr2=0x5651728ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 -- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 msgr2=0x5651728ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 0x5651728ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 -- 172.21.15.32:0/3378960461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651728ab000 con 0x565172899520 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa136ffd640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 0x5651728ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa1377fe640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5651728bb900 tx=0x7fa128005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2965e7bd9d0ba30a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa116ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128020670 con 0x565172899520 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa128020810 con 0x565172899520 2024-04-23T18:16:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa12c0043d0 con 0x565172899520 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa128029430 con 0x565172899520 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa140b96740 10 client.15651 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56517291f0a0 con 0x565172899520 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.563+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12c0043d0 con 0x565172899520 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa1280326a0 con 0x565172899520 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 10 client.15651 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 10 client.15651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 20 client.15651 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56517291f810 0x565172921c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 4 client.15651 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565172806200 con 0x56517291f810 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa136ffd640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56517291f810 0x565172921c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa136ffd640 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56517291f810 0x565172921c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79ff965d0dd4d344 server_cookie=5ae0f42b61fa3ae4 in_seq=0 out_seq=0 2024-04-23T18:16:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:02.564+0000 7fa116ffd640 10 client.15651 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:02.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:02 smithi032 ceph-mon[29221]: pgmap v30: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 MiB/s wr, 29 op/s 2024-04-23T18:16:02.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:02 smithi032 ceph-mon[29221]: mgrmap e24: x(active, since 54s), standbys: z, y 2024-04-23T18:16:02.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:02 smithi040 ceph-mon[32626]: pgmap v30: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 MiB/s wr, 29 op/s 2024-04-23T18:16:02.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:02 smithi040 ceph-mon[32626]: mgrmap e24: x(active, since 54s), standbys: z, y 2024-04-23T18:16:02.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:02 smithi064 ceph-mon[33062]: pgmap v30: 97 pgs: 97 active+clean; 858 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 MiB/s wr, 29 op/s 2024-04-23T18:16:02.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:02 smithi064 ceph-mon[33062]: mgrmap e24: x(active, since 54s), standbys: z, y 2024-04-23T18:16:03.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:03.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 -- 172.21.15.32:0/3717032415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 msgr2=0x7fcf24101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/3717032415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24101500 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009900 tx=0x7fcf1402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 -- 172.21.15.32:0/3717032415 shutdown_connections 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/3717032415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 0x7fcf2410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/3717032415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24101500 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/3717032415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 0x7fcf241072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 -- 172.21.15.32:0/3717032415 >> 172.21.15.32:0/3717032415 conn(0x7fcf24077c40 msgr2=0x7fcf24078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.527+0000 7fcf2ae20640 1 -- 172.21.15.32:0/3717032415 shutdown_connections 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 -- 172.21.15.32:0/3717032415 wait complete. 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 Processor -- start 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 -- start start 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 0x7fcf24198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 0x7fcf2419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf24112250 con 0x7fcf241010a0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf241120d0 con 0x7fcf24101a40 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf2ae20640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcf241123d0 con 0x7fcf24106ed0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf28bbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf28bbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59686/0 (socket says 172.21.15.32:59686) 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.528+0000 7fcf28bbc640 1 -- 172.21.15.32:0/73544485 learned_addr learned my addr 172.21.15.32:0/73544485 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf1bfff640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 0x7fcf24198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf293bd640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 0x7fcf2419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 -- 172.21.15.32:0/73544485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 msgr2=0x7fcf2419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 0x7fcf2419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 -- 172.21.15.32:0/73544485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 msgr2=0x7fcf24198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 0x7fcf24198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 -- 172.21.15.32:0/73544485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2419d700 con 0x7fcf241010a0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf28bbc640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000e9f0 tx=0x7fcf2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2000cd80 con 0x7fcf241010a0 2024-04-23T18:16:03.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf200108e0 con 0x7fcf241010a0 2024-04-23T18:16:03.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2419d9f0 con 0x7fcf241010a0 2024-04-23T18:16:03.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf20018980 con 0x7fcf241010a0 2024-04-23T18:16:03.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.529+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf241a5280 con 0x7fcf241010a0 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.530+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf24068e00 con 0x7fcf241010a0 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.531+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fcf20010430 con 0x7fcf241010a0 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.532+0000 7fcf19ffb640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 0x7fcf0c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.532+0000 7fcf1bfff640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 0x7fcf0c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.532+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fcf200d0d90 con 0x7fcf241010a0 2024-04-23T18:16:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.533+0000 7fcf1bfff640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 0x7fcf0c0a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf14000c00 tx=0x7fcf14031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.537+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf2009d230 con 0x7fcf241010a0 2024-04-23T18:16:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.765+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf24109370 con 0x7fcf241010a0 2024-04-23T18:16:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.766+0000 7fcf19ffb640 1 -- 172.21.15.32:0/73544485 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11854 (secure 0 0 0) 0x7fcf2009cb30 con 0x7fcf241010a0 2024-04-23T18:16:03.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:16:03.760 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:30.858665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:16:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 msgr2=0x7fcf0c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 0x7fcf0c0a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf14000c00 tx=0x7fcf14031400 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 msgr2=0x7fcf24198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000e9f0 tx=0x7fcf2000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 shutdown_connections 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf24106ed0 0x7fcf2419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf24101a40 0x7fcf24198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcf0c0a3ec0 0x7fcf0c0a6380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 --2- 172.21.15.32:0/73544485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf241010a0 0x7fcf24198710 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 >> 172.21.15.32:0/73544485 conn(0x7fcf24077c40 msgr2=0x7fcf240ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 shutdown_connections 2024-04-23T18:16:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.768+0000 7fcf2ae20640 1 -- 172.21.15.32:0/73544485 wait complete. 2024-04-23T18:16:03.773 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id ef875bc0 --release --await' 2024-04-23T18:16:03.773 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id ef875bc0 --release --await 2024-04-23T18:16:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.810+0000 7fa116ffd640 1 -- 172.21.15.32:0/3378960461 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565172806200 con 0x56517291f810 2024-04-23T18:16:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.810+0000 7fa116ffd640 10 client.15651 handle_command_reply: tid=0 2024-04-23T18:16:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.810+0000 7fa140b96740 1 client.15651 shutdown 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 msgr2=0x5651728b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5651728bb900 tx=0x7fa128005c60 comp rx=0 tx=0).stop 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 shutdown_connections 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56517291f810 0x565172921c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5651728a8c10 0x5651728ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa116ffd640 0 client.15651 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565172899e40 0x5651728ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 --2- 172.21.15.32:0/3378960461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565172899520 0x5651728b0fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 >> 172.21.15.32:0/3378960461 conn(0x565172895b30 msgr2=0x5651728a0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 shutdown_connections 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 1 -- 172.21.15.32:0/3378960461 wait complete. 2024-04-23T18:16:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa140b96740 20 client.15651 trim_cache size 0 max 16384 2024-04-23T18:16:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:16:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:16:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:16:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 msgr2=0x7fa1200a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 0x7fa1200a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c032c60 comp rx=0 tx=0).stop 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 msgr2=0x7fa13819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa13000eaa0 tx=0x7fa13000ef60 comp rx=0 tx=0).stop 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 shutdown_connections 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa1200a3e30 0x7fa1200a62f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa13819cfb0 0x7fa13819d460 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1381059c0 0x7fa138198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 --2- 172.21.15.32:0/939065629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa138100af0 0x7fa138198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.811+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 >> 172.21.15.32:0/939065629 conn(0x7fa138076200 msgr2=0x7fa1381045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.812+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 shutdown_connections 2024-04-23T18:16:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:03.812+0000 7fa13dda2640 1 -- 172.21.15.32:0/939065629 wait complete. 2024-04-23T18:16:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3915014835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec106160 msgr2=0x7fa7ec10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3915014835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec106160 0x7fa7ec10dd30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009ca0 tx=0x7fa7e402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3915014835 shutdown_connections 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3915014835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec106160 0x7fa7ec10dd30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3915014835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec101a40 0x7fa7ec105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3915014835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 0x7fa7ec101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3915014835 >> 172.21.15.32:0/3915014835 conn(0x7fa7ec077fd0 msgr2=0x7fa7ec0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3915014835 shutdown_connections 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3915014835 wait complete. 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 Processor -- start 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.891+0000 7fa7f3a7c640 1 -- start start 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 0x7fa7ec1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec106160 0x7fa7ec198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1110b0 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec110f30 con 0x7fa7ec101090 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec111230 con 0x7fa7ec106160 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59706/0 (socket says 172.21.15.32:59706) 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 -- 172.21.15.32:0/3289287568 learned_addr learned my addr 172.21.15.32:0/3289287568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec106160 0x7fa7ec198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f1818640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 0x7fa7ec1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 -- 172.21.15.32:0/3289287568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec106160 msgr2=0x7fa7ec198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec106160 0x7fa7ec198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 -- 172.21.15.32:0/3289287568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 msgr2=0x7fa7ec1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 0x7fa7ec1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 -- 172.21.15.32:0/3289287568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec19daa0 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f2019640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005cc0 tx=0x7fa7e402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7e2ffd640 1 -- 172.21.15.32:0/3289287568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4003bf0 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7e2ffd640 1 -- 172.21.15.32:0/3289287568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e4003d90 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec19dd30 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.892+0000 7fa7e2ffd640 1 -- 172.21.15.32:0/3289287568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e402fbc0 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.893+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ec1a5550 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7e2ffd640 1 -- 172.21.15.32:0/3289287568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fa7e402fd60 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 Processor -- start 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 -- start start 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c70870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c70db0 0x5616a9c7af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616a9bd56d0 con 0x5616a9c7b480 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5616a9b721a0 con 0x5616a9c70db0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7f6798740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5616a9bd1690 con 0x5616a9c70490 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.895+0000 7fa7e2ffd640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 0x7fa7d80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59714/0 (socket says 172.21.15.32:59714) 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 -- 172.21.15.32:0/650833228 learned_addr learned my addr 172.21.15.32:0/650833228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c70870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1818640 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c70db0 0x5616a9c7af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f2019640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 0x7fa7d80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 -- 172.21.15.32:0/650833228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 msgr2=0x5616a9c70870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c70870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 -- 172.21.15.32:0/650833228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c70db0 msgr2=0x5616a9c7af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c70db0 0x5616a9c7af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7e2ffd640 1 -- 172.21.15.32:0/3289287568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa7e40f2070 con 0x7fa7ec19cfb0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 -- 172.21.15.32:0/650833228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616a9b7c5b0 con 0x5616a9c7b480 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f1017640 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc005c50 tx=0x7fa7dc00a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae591d37ffd27f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7f2019640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 0x7fa7d80a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40ef800 tx=0x7fa7e4090a10 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7d7fff640 1 -- 172.21.15.32:0/650833228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc004020 con 0x5616a9c7b480 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7d7fff640 1 -- 172.21.15.32:0/650833228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7dc0054d0 con 0x5616a9c7b480 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.896+0000 7fa7d7fff640 1 -- 172.21.15.32:0/650833228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc004af0 con 0x5616a9c7b480 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- 172.21.15.32:0/650833228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 msgr2=0x5616a9c7d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc005c50 tx=0x7fa7dc00a600 comp rx=0 tx=0).stop 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- 172.21.15.32:0/650833228 shutdown_connections 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c7b480 0x5616a9c7d870 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c70db0 0x5616a9c7af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 --2- 172.21.15.32:0/650833228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c70870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- 172.21.15.32:0/650833228 >> 172.21.15.32:0/650833228 conn(0x5616a9c6c7b0 msgr2=0x5616a9c6cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- 172.21.15.32:0/650833228 shutdown_connections 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- 172.21.15.32:0/650833228 wait complete. 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.897+0000 7fa7f6798740 1 -- start start 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 Processor -- start 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 0x5616a9c781d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c788f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616a9c7e310 con 0x5616a9c70490 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5616a9b6fb20 con 0x5616a9c7b480 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5616a9c7e490 con 0x5616a9c70db0 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f6798740 10 client.0 _ll_register_callbacks cb 0x5616a9b6f530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59728/0 (socket says 172.21.15.32:59728) 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 -- 172.21.15.32:0/868746449 learned_addr learned my addr 172.21.15.32:0/868746449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1818640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 0x5616a9c781d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 -- 172.21.15.32:0/868746449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 msgr2=0x5616a9c781d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 0x5616a9c781d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 -- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 msgr2=0x5616a9c788f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 -- 172.21.15.32:0/868746449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616a98e7920 con 0x5616a9c70490 2024-04-23T18:16:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1818640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 0x5616a9c781d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7f1017640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc005410 tx=0x7fa7dc005440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f76f7e7993fd367 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.898+0000 7fa7d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc024030 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa7dc004db0 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7dc0054d0 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa7dc02d020 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 10 client.15669 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5616a99b3d30 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7dc0054d0 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa7dc01e590 con 0x5616a9c70490 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 10 client.15669 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 10 client.15669 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 20 client.15669 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9c8b5f0 0x5616a9c8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 4 client.15669 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.899+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616a9b93500 con 0x5616a9c8b5f0 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.900+0000 7fa7f1818640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9c8b5f0 0x5616a9c8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.900+0000 7fa7f1818640 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9c8b5f0 0x5616a9c8d9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=808f86f31b8daaa8 server_cookie=def2c08eaaba58c in_seq=0 out_seq=0 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.900+0000 7fa7d57fa640 10 client.15669 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7d57fa640 1 -- 172.21.15.32:0/868746449 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616a9b93500 con 0x5616a9c8b5f0 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7d57fa640 10 client.15669 handle_command_reply: tid=0 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 client.15669 shutdown 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 msgr2=0x5616a9c7aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc005410 tx=0x7fa7dc005440 comp rx=0 tx=0).stop 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 shutdown_connections 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9c8b5f0 0x5616a9c8d9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c788f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70db0 0x5616a9c781d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7d57fa640 0 client.15669 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 --2- 172.21.15.32:0/868746449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70490 0x5616a9c7aee0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 >> 172.21.15.32:0/868746449 conn(0x5616a9c6c7b0 msgr2=0x5616a9c725b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 shutdown_connections 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 1 -- 172.21.15.32:0/868746449 wait complete. 2024-04-23T18:16:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.901+0000 7fa7f6798740 20 client.15669 trim_cache size 0 max 16384 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.918+0000 7fa7f6798740 1 Processor -- start 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 -- start start 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c87200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c93210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1b02e0 con 0x5616a9c70db0 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1adbb0 con 0x5616a9c7b480 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f6798740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1aba20 con 0x5616a9c70490 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59740/0 (socket says 172.21.15.32:59740) 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 -- 172.21.15.32:0/1555657333 learned_addr learned my addr 172.21.15.32:0/1555657333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1017640 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c87200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1017640 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c93210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 -- 172.21.15.32:0/1555657333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 msgr2=0x5616a9c87200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c87200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 -- 172.21.15.32:0/1555657333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 msgr2=0x5616a9c93210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c93210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 -- 172.21.15.32:0/1555657333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616a9c70be0 con 0x5616a9c70db0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7f1818640 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x5616a9c8e1b0 tx=0x7fa7e800b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aed8c9017995304 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7d77fe640 1 -- 172.21.15.32:0/1555657333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8004020 con 0x5616a9c70db0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7d77fe640 1 -- 172.21.15.32:0/1555657333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e80041c0 con 0x5616a9c70db0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.919+0000 7fa7d77fe640 1 -- 172.21.15.32:0/1555657333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e801b900 con 0x5616a9c70db0 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- 172.21.15.32:0/1555657333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 msgr2=0x5616a9c7aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x5616a9c8e1b0 tx=0x7fa7e800b4d0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- 172.21.15.32:0/1555657333 shutdown_connections 2024-04-23T18:16:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c93210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7aa10 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 --2- 172.21.15.32:0/1555657333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c87200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- 172.21.15.32:0/1555657333 >> 172.21.15.32:0/1555657333 conn(0x5616a9c6c7b0 msgr2=0x5616a9c71be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- 172.21.15.32:0/1555657333 shutdown_connections 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- 172.21.15.32:0/1555657333 wait complete. 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 -- start start 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.920+0000 7fa7f6798740 1 Processor -- start 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c83910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c7ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1ac070 con 0x5616a9c70db0 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1ad4e0 con 0x5616a9c7b480 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec1ac740 con 0x5616a9c70490 2024-04-23T18:16:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 10 client.0 _ll_register_callbacks cb 0x7fa7dc011590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59752/0 (socket says 172.21.15.32:59752) 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 -- 172.21.15.32:0/3229089371 learned_addr learned my addr 172.21.15.32:0/3229089371 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c7ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c83910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 -- 172.21.15.32:0/3229089371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 msgr2=0x5616a9c83910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c83910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 -- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 msgr2=0x5616a9c7ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c7ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 -- 172.21.15.32:0/3229089371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616a9c7f180 con 0x5616a9c70db0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c7ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f1818640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800b370 tx=0x7fa7e80142f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d599d74077a1a5bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8014a00 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7e80041c0 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.921+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa7e8014ba0 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa7e8014da0 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 10 client.15675 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5616a9cf2860 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e80041c0 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa7e802c030 con 0x5616a9c70db0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 10 client.15675 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 10 client.15675 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 20 client.15675 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9cf2f10 0x5616a9cf5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 4 client.15675 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ef875bc0", "release": true, "await": true}] 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ef875bc0", "release": true, "await": true}) -- 0x5616a9be30f0 con 0x5616a9cf2f10 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.922+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9cf2f10 0x5616a9cf5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.923+0000 7fa7f1017640 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9cf2f10 0x5616a9cf5330 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f579302a78b179b7 server_cookie=472d203eaa8c9d2c in_seq=0 out_seq=0 2024-04-23T18:16:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.923+0000 7fa7d57fa640 10 client.15675 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7d57fa640 1 -- 172.21.15.32:0/3229089371 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5616a9be30f0 con 0x5616a9cf2f10 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7d57fa640 10 client.15675 handle_command_reply: tid=0 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 client.15675 shutdown 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 msgr2=0x5616a9c7e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800b370 tx=0x7fa7e80142f0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 shutdown_connections 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616a9cf2f10 0x5616a9cf5330 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616a9c7b480 0x5616a9c7ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616a9c70db0 0x5616a9c7e6d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 --2- 172.21.15.32:0/3229089371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616a9c70490 0x5616a9c83910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7d57fa640 0 client.15675 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.960+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 >> 172.21.15.32:0/3229089371 conn(0x5616a9c6c7b0 msgr2=0x5616a9c72470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 shutdown_connections 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f6798740 1 -- 172.21.15.32:0/3229089371 wait complete. 2024-04-23T18:16:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f6798740 20 client.15675 trim_cache size 0 max 16384 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 24, 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 4, 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "ef875bc0": { 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 4, 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:16:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:03.957 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:16:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 msgr2=0x7fa7d80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 0x7fa7d80a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40ef800 tx=0x7fa7e4090a10 comp rx=0 tx=0).stop 2024-04-23T18:16:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 msgr2=0x7fa7ec19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005cc0 tx=0x7fa7e402f380 comp rx=0 tx=0).stop 2024-04-23T18:16:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 shutdown_connections 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa7d80a3e40 0x7fa7d80a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ec19cfb0 0x7fa7ec19d460 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7ec106160 0x7fa7ec198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 --2- 172.21.15.32:0/3289287568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec101090 0x7fa7ec1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.961+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 >> 172.21.15.32:0/3289287568 conn(0x7fa7ec077fd0 msgr2=0x7fa7ec06aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.962+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 shutdown_connections 2024-04-23T18:16:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:03.962+0000 7fa7f3a7c640 1 -- 172.21.15.32:0/3289287568 wait complete. 2024-04-23T18:16:03.974 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ef875bc0', seconds elapsed: 32.4 2024-04-23T18:16:03.974 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:16:03.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 -- 172.21.15.32:0/3904039205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 msgr2=0x7fdac410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/3904039205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac410ee40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009ca0 tx=0x7fdabc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 -- 172.21.15.32:0/3904039205 shutdown_connections 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/3904039205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac410ee40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/3904039205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac4102b50 0x7fdac4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/3904039205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac41021a0 0x7fdac4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 -- 172.21.15.32:0/3904039205 >> 172.21.15.32:0/3904039205 conn(0x7fdac40fda40 msgr2=0x7fdac40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.141+0000 7fdacbb8a640 1 -- 172.21.15.32:0/3904039205 shutdown_connections 2024-04-23T18:16:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 -- 172.21.15.32:0/3904039205 wait complete. 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 Processor -- start 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 -- start start 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac419cfd0 0x7fdac419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdac4112230 con 0x7fdac4107270 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41120b0 con 0x7fdac41021a0 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.142+0000 7fdacbb8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41123b0 con 0x7fdac419cfd0 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9125640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:44790/0 (socket says 172.21.15.32:44790) 2024-04-23T18:16:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 -- 172.21.15.32:0/1014418853 learned_addr learned my addr 172.21.15.32:0/1014418853 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdaca127640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac419cfd0 0x7fdac419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 -- 172.21.15.32:0/1014418853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac419cfd0 msgr2=0x7fdac419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac419cfd0 0x7fdac419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 -- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 msgr2=0x7fdac4198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac4198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 -- 172.21.15.32:0/1014418853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac419dac0 con 0x7fdac41021a0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdac9926640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdac000e9f0 tx=0x7fdac000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac000cd80 con 0x7fdac41021a0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac419ddb0 con 0x7fdac41021a0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac00054d0 con 0x7fdac41021a0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.143+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0010690 con 0x7fdac41021a0 2024-04-23T18:16:04.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.144+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac41a55a0 con 0x7fdac41021a0 2024-04-23T18:16:04.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.145+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac4068e00 con 0x7fdac41021a0 2024-04-23T18:16:04.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.148+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fdac0004e90 con 0x7fdac41021a0 2024-04-23T18:16:04.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.149+0000 7fdabaffd640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 0x7fdab00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.149+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fdac00d0ec0 con 0x7fdac41021a0 2024-04-23T18:16:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.149+0000 7fdac9125640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 0x7fdab00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.150+0000 7fdac9125640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 0x7fdab00a6380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdac41087f0 tx=0x7fdab4005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.151+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdac009d3c0 con 0x7fdac41021a0 2024-04-23T18:16:04.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.528+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdac410a4d0 con 0x7fdac41021a0 2024-04-23T18:16:04.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.529+0000 7fdabaffd640 1 -- 172.21.15.32:0/1014418853 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11854 (secure 0 0 0) 0x7fdac009ccc0 con 0x7fdac41021a0 2024-04-23T18:16:04.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:16:04.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:15:30.858665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1625538187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1625538187},{"type":"v1","addr":"172.21.15.64:6837","nonce":1625538187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:16:04.524 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-04-23T18:16:04.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 msgr2=0x7fdab00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 0x7fdab00a6380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdac41087f0 tx=0x7fdab4005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 msgr2=0x7fdac4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdac000e9f0 tx=0x7fdac000edf0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 shutdown_connections 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac419cfd0 0x7fdac419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdab00a3ec0 0x7fdab00a6380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4107270 0x7fdac4198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 --2- 172.21.15.32:0/1014418853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac41021a0 0x7fdac4198710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 >> 172.21.15.32:0/1014418853 conn(0x7fdac40fda40 msgr2=0x7fdac40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 shutdown_connections 2024-04-23T18:16:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.533+0000 7fdacbb8a640 1 -- 172.21.15.32:0/1014418853 wait complete. 2024-04-23T18:16:04.544 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:16:04.544 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:16:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 -- 172.21.15.32:0/2846943924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b4101070 msgr2=0x7f83b4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 --2- 172.21.15.32:0/2846943924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b4101070 0x7f83b4106ba0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f839c009860 tx=0x7f839c02a340 comp rx=0 tx=0).stop 2024-04-23T18:16:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 -- 172.21.15.32:0/2846943924 shutdown_connections 2024-04-23T18:16:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 --2- 172.21.15.32:0/2846943924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b4107280 0x7f83b410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 --2- 172.21.15.32:0/2846943924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b4101070 0x7f83b4106ba0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 --2- 172.21.15.32:0/2846943924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b41006c0 0x7f83b4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 -- 172.21.15.32:0/2846943924 >> 172.21.15.32:0/2846943924 conn(0x7f83b40773a0 msgr2=0x7f83b40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.699+0000 7f83b9515640 1 -- 172.21.15.32:0/2846943924 shutdown_connections 2024-04-23T18:16:04.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.700+0000 7f83b9515640 1 -- 172.21.15.32:0/2846943924 wait complete. 2024-04-23T18:16:04.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.701+0000 7f83b9515640 1 Processor -- start 2024-04-23T18:16:04.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.701+0000 7f83b9515640 1 -- start start 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 0x7f83b419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 0x7f83b419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b41985f0 0x7f83b4198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83b41121a0 con 0x7f83b41006c0 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4112020 con 0x7f83b41985f0 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b9515640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4112320 con 0x7f83b4107280 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 0x7f83b419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 0x7f83b419e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59788/0 (socket says 172.21.15.32:59788) 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b2ffd640 1 -- 172.21.15.32:0/1403495767 learned_addr learned my addr 172.21.15.32:0/1403495767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 0x7f83b419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.702+0000 7f83b37fe640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b41985f0 0x7f83b4198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 -- 172.21.15.32:0/1403495767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b41985f0 msgr2=0x7f83b4198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b41985f0 0x7f83b4198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 -- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 msgr2=0x7f83b419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 0x7f83b419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 -- 172.21.15.32:0/1403495767 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b4199110 con 0x7f83b4107280 2024-04-23T18:16:04.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.703+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 0x7f83b419e980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f839c02a860 tx=0x7f839c032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.708+0000 7f83abfff640 1 -- 172.21.15.32:0/1403495767 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c037040 con 0x7f83b4107280 2024-04-23T18:16:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.708+0000 7f83abfff640 1 -- 172.21.15.32:0/1403495767 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f839c030c10 con 0x7f83b4107280 2024-04-23T18:16:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.708+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b41993a0 con 0x7f83b4107280 2024-04-23T18:16:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.708+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b41a5530 con 0x7f83b4107280 2024-04-23T18:16:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.708+0000 7f83abfff640 1 -- 172.21.15.32:0/1403495767 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c0035f0 con 0x7f83b4107280 2024-04-23T18:16:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.713+0000 7f83bc220740 1 Processor -- start 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 -- start start 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608e160 0x55d32608e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 0x55d326089e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 0x55d3260960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d325fe5370 con 0x55d32608a380 2024-04-23T18:16:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d325f822e0 con 0x55d32608e160 2024-04-23T18:16:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.714+0000 7f83bc220740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d325fe1330 con 0x55d3260899c0 2024-04-23T18:16:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.715+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 0x55d326089e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.715+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 0x55d326089e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:47526/0 (socket says 172.21.15.32:47526) 2024-04-23T18:16:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.715+0000 7f83b2ffd640 1 -- 172.21.15.32:0/448958662 learned_addr learned my addr 172.21.15.32:0/448958662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.716+0000 7f83b37fe640 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608e160 0x55d32608e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83abfff640 1 -- 172.21.15.32:0/1403495767 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7f839c043030 con 0x7f83b4107280 2024-04-23T18:16:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 0x55d3260960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 -- 172.21.15.32:0/448958662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 msgr2=0x55d326089e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 0x55d326089e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 -- 172.21.15.32:0/448958662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608e160 msgr2=0x55d32608e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608e160 0x55d32608e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.717+0000 7f83b27fc640 1 -- 172.21.15.32:0/448958662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d325dc3d80 con 0x55d32608a380 2024-04-23T18:16:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.718+0000 7f83abfff640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 0x7f83a00a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.718+0000 7f83abfff640 1 -- 172.21.15.32:0/1403495767 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f839c0ef6a0 con 0x7f83b4107280 2024-04-23T18:16:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.719+0000 7f83b27fc640 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 0x55d3260960b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f839c0ecd70 tx=0x7f839c0f5ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d0dfc87ce7a2ce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.720+0000 7f83a97fa640 1 -- 172.21.15.32:0/448958662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c0b7810 con 0x55d32608a380 2024-04-23T18:16:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.720+0000 7f83a97fa640 1 -- 172.21.15.32:0/448958662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f839c030450 con 0x55d32608a380 2024-04-23T18:16:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.720+0000 7f83a97fa640 1 -- 172.21.15.32:0/448958662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c0baed0 con 0x55d32608a380 2024-04-23T18:16:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.721+0000 7f83bc220740 1 -- 172.21.15.32:0/448958662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 msgr2=0x55d3260960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.721+0000 7f83bc220740 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 0x55d3260960b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f839c0ecd70 tx=0x7f839c0f5ac0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.721+0000 7f83b37fe640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 0x7f83a00a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83bc220740 1 -- 172.21.15.32:0/448958662 shutdown_connections 2024-04-23T18:16:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83bc220740 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608a380 0x55d3260960b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83bc220740 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d3260899c0 0x55d326089e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83bc220740 1 --2- 172.21.15.32:0/448958662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608e160 0x55d32608e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83bc220740 1 -- 172.21.15.32:0/448958662 >> 172.21.15.32:0/448958662 conn(0x55d32607be70 msgr2=0x55d32607c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.722+0000 7f83b37fe640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 0x7f83a00a64d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83ac005fd0 tx=0x7f83ac009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.724+0000 7f83bc220740 1 -- 172.21.15.32:0/448958662 shutdown_connections 2024-04-23T18:16:04.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.726+0000 7f83bc220740 1 -- 172.21.15.32:0/448958662 wait complete. 2024-04-23T18:16:04.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.726+0000 7f83bc220740 1 -- start start 2024-04-23T18:16:04.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.726+0000 7f83bc220740 1 Processor -- start 2024-04-23T18:16:04.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 0x55d3260939e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 0x55d326093f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d32608ece0 con 0x55d3260899c0 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d325f7ffb0 con 0x55d32608a380 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d3260965f0 con 0x55d32608e160 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83bc220740 10 client.0 _ll_register_callbacks cb 0x55d325f7f9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59808/0 (socket says 172.21.15.32:59808) 2024-04-23T18:16:04.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 -- 172.21.15.32:0/1400831767 learned_addr learned my addr 172.21.15.32:0/1400831767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b37fe640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 0x55d326093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 0x55d3260939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 -- 172.21.15.32:0/1400831767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 msgr2=0x55d326093f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 0x55d326093f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 -- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 msgr2=0x55d3260939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 0x55d3260939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b2ffd640 1 -- 172.21.15.32:0/1400831767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d325e24510 con 0x55d3260899c0 2024-04-23T18:16:04.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b37fe640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 0x55d326093f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:04.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.727+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 0x55d3260939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.728+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f83a40126f0 tx=0x7f83a4012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e590ed0c8535a8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.728+0000 7f839a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a400f920 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f83a4012e00 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d325ff2d90 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f83a400fac0 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f83bc220740 10 client.15693 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.729+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d325cb7670 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d325ff2d90 con 0x55d3260899c0 2024-04-23T18:16:04.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f83a4004920 con 0x55d3260899c0 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 10 client.15693 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 10 client.15693 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 20 client.15693 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d3260b4940 0x55d3260b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 4 client.15693 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.730+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d325fecc00 con 0x55d3260b4940 2024-04-23T18:16:04.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.731+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d3260b4940 0x55d3260b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.734+0000 7f83b27fc640 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d3260b4940 0x55d3260b6db0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f247ec11d69b3ddc server_cookie=1acf277dc1174193 in_seq=0 out_seq=0 2024-04-23T18:16:04.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.734+0000 7f839a7fc640 10 client.15693 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.738+0000 7f839a7fc640 1 -- 172.21.15.32:0/1400831767 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d325fecc00 con 0x55d3260b4940 2024-04-23T18:16:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.738+0000 7f839a7fc640 10 client.15693 handle_command_reply: tid=0 2024-04-23T18:16:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.738+0000 7f83bc220740 1 client.15693 shutdown 2024-04-23T18:16:04.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.740+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 msgr2=0x55d3260934a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.740+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f83a40126f0 tx=0x7f83a4012bb0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 shutdown_connections 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d3260b4940 0x55d3260b6db0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608e160 0x55d326093f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d32608a380 0x55d3260939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 --2- 172.21.15.32:0/1400831767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d3260899c0 0x55d3260934a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 >> 172.21.15.32:0/1400831767 conn(0x55d32607be70 msgr2=0x55d326089240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.743+0000 7f839a7fc640 0 client.15693 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:04.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.744+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 shutdown_connections 2024-04-23T18:16:04.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.745+0000 7f83bc220740 1 -- 172.21.15.32:0/1400831767 wait complete. 2024-04-23T18:16:04.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.747+0000 7f83bc220740 20 client.15693 trim_cache size 0 max 16384 2024-04-23T18:16:04.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:04 smithi032 ceph-mon[29221]: pgmap v31: 97 pgs: 97 active+clean; 858 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 48 op/s 2024-04-23T18:16:04.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:04 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/73544485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:04.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 Processor -- start 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 -- start start 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d326091050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8394010f60 con 0x55d32608e160 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f839400e830 con 0x55d3260899c0 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.788+0000 7f83bc220740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f839400c680 con 0x55d32608a380 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d326091050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59822/0 (socket says 172.21.15.32:59822) 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d326091050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:47544/0 (socket says 172.21.15.32:47544) 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 -- 172.21.15.32:0/764150843 learned_addr learned my addr 172.21.15.32:0/764150843 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 -- 172.21.15.32:0/764150843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 msgr2=0x55d326091050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d326091050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 -- 172.21.15.32:0/764150843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 msgr2=0x55d32609f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 -- 172.21.15.32:0/764150843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d326092450 con 0x55d32608e160 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83b37fe640 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f83ac01e850 tx=0x7f83ac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d669715bef9b73c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83a8ff9640 1 -- 172.21.15.32:0/764150843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac01e0d0 con 0x55d32608e160 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83a8ff9640 1 -- 172.21.15.32:0/764150843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac003e20 con 0x55d32608e160 2024-04-23T18:16:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.789+0000 7f83a8ff9640 1 -- 172.21.15.32:0/764150843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac004020 con 0x55d32608e160 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.790+0000 7f83bc220740 1 -- 172.21.15.32:0/764150843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 msgr2=0x55d3260b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.790+0000 7f83bc220740 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f83ac01e850 tx=0x7f83ac016000 comp rx=0 tx=0).stop 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 -- 172.21.15.32:0/764150843 shutdown_connections 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260b68e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d326091050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 --2- 172.21.15.32:0/764150843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 -- 172.21.15.32:0/764150843 >> 172.21.15.32:0/764150843 conn(0x55d32607be70 msgr2=0x55d32608ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 -- 172.21.15.32:0/764150843 shutdown_connections 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.791+0000 7f83bc220740 1 -- 172.21.15.32:0/764150843 wait complete. 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.792+0000 7f83bc220740 1 -- start start 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.792+0000 7f83bc220740 1 Processor -- start 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.792+0000 7f83bc220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d32609bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260901a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f839400ccf0 con 0x55d32608e160 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f839400e160 con 0x55d3260899c0 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f839400d3c0 con 0x55d32608a380 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83bc220740 10 client.0 _ll_register_callbacks cb 0x7f83a4009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:44848/0 (socket says 172.21.15.32:44848) 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 -- 172.21.15.32:0/756229367 learned_addr learned my addr 172.21.15.32:0/756229367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b37fe640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260901a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b27fc640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d32609bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 -- 172.21.15.32:0/756229367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 msgr2=0x55d32609bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d32609bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 -- 172.21.15.32:0/756229367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 msgr2=0x55d3260901a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260901a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.793+0000 7f83b2ffd640 1 -- 172.21.15.32:0/756229367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d326090760 con 0x55d3260899c0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.794+0000 7f83b2ffd640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83a4005c80 tx=0x7f83a4004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bca89156cb367f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.794+0000 7f839a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.794+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4004a00 con 0x55d3260899c0 2024-04-23T18:16:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.794+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83ac003e20 con 0x55d3260899c0 2024-04-23T18:16:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.795+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f83a4004f10 con 0x55d3260899c0 2024-04-23T18:16:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f83a4005110 con 0x55d3260899c0 2024-04-23T18:16:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 10 client.25346 fetch_fsmap learned FSMap version 24 2024-04-23T18:16:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d32611b7e0 con 0x55d3260899c0 2024-04-23T18:16:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac003e20 con 0x55d3260899c0 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f83a4026af0 con 0x55d3260899c0 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 10 client.25346 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 10 client.25346 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 20 client.25346 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d32611c0b0 0x55d32611e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 4 client.25346 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55d325f68ca0 con 0x55d32611c0b0 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.796+0000 7f83b27fc640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d32611c0b0 0x55d32611e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.797+0000 7f839a7fc640 10 client.25346 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.797+0000 7f83b27fc640 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d32611c0b0 0x55d32611e4d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=810e3b42967278a5 server_cookie=a7d0ce32dbe22b6c in_seq=0 out_seq=0 2024-04-23T18:16:04.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:04 smithi040 ceph-mon[32626]: pgmap v31: 97 pgs: 97 active+clean; 858 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 48 op/s 2024-04-23T18:16:04.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:04 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/73544485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f839a7fc640 1 -- 172.21.15.32:0/756229367 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55d325f68ca0 con 0x55d32611c0b0 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f839a7fc640 10 client.25346 handle_command_reply: tid=0 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 client.25346 shutdown 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 msgr2=0x55d32609b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83a4005c80 tx=0x7f83a4004600 comp rx=0 tx=0).stop 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 shutdown_connections 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d32611c0b0 0x55d32611e4d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d32608e160 0x55d3260901a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f839a7fc640 0 client.25346 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d32608a380 0x55d32609bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 --2- 172.21.15.32:0/756229367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3260899c0 0x55d32609b840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 >> 172.21.15.32:0/756229367 conn(0x55d32607be70 msgr2=0x55d326089240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 shutdown_connections 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.886+0000 7f83bc220740 1 -- 172.21.15.32:0/756229367 wait complete. 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83bc220740 20 client.25346 trim_cache size 0 max 16384 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 24, 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 6, 2024-04-23T18:16:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "61826c57": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 6, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:04.884 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 msgr2=0x7f83a00a64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 0x7f83a00a64d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83ac005fd0 tx=0x7f83ac009500 comp rx=0 tx=0).stop 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 msgr2=0x7f83b419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 0x7f83b419e980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f839c02a860 tx=0x7f839c032d50 comp rx=0 tx=0).stop 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 shutdown_connections 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b41985f0 0x7f83b4198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83b4107280 0x7f83b419e980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83a00a4010 0x7f83a00a64d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 --2- 172.21.15.32:0/1403495767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b41006c0 0x7f83b419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 >> 172.21.15.32:0/1403495767 conn(0x7f83b40773a0 msgr2=0x7f83b4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 shutdown_connections 2024-04-23T18:16:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:04.887+0000 7f83b9515640 1 -- 172.21.15.32:0/1403495767 wait complete. 2024-04-23T18:16:04.893 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '61826c57', quiesce took 0.9 seconds. Will release after: 30.0 2024-04-23T18:16:04.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-mon[33062]: pgmap v31: 97 pgs: 97 active+clean; 858 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 48 op/s 2024-04-23T18:16:04.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/73544485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: *** Caught signal (Segmentation fault) ** 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: in thread 7f3027859700 thread_name:md_log_replay 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3038c1ad20] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2: /usr/bin/ceph-mds(+0x549372) [0x5637593b2372] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5637593b5b23] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 4: (MDLog::_replay_thread()+0x617) [0x563759410647] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563759419d31] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f3038c101ca] 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 7: clone() 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.592+0000 7f3027859700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:16:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: in thread 7f3027859700 thread_name:md_log_replay 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3038c1ad20] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2: /usr/bin/ceph-mds(+0x549372) [0x5637593b2372] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5637593b5b23] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 4: (MDLog::_replay_thread()+0x617) [0x563759410647] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563759419d31] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f3038c101ca] 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 7: clone() 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 0> 2024-04-23T18:16:04.592+0000 7f3027859700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:16:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: in thread 7f3027859700 thread_name:md_log_replay 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3038c1ad20] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2: /usr/bin/ceph-mds(+0x549372) [0x5637593b2372] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5637593b5b23] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 4: (MDLog::_replay_thread()+0x617) [0x563759410647] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563759419d31] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f3038c101ca] 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 7: clone() 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 0> 2024-04-23T18:16:04.592+0000 7f3027859700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: in thread 7f3027859700 thread_name:md_log_replay 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3038c1ad20] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2: /usr/bin/ceph-mds(+0x549372) [0x5637593b2372] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5637593b5b23] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 4: (MDLog::_replay_thread()+0x617) [0x563759410647] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563759419d31] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f3038c101ca] 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 7: clone() 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[64243]: 2024-04-23T18:16:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:04 smithi064 podman[67449]: 2024-04-23 18:16:04.661226424 +0000 UTC m=+0.023780599 container died b5a8e1bcf9a096516292928af5c02c0311fa30fd1db5da8c3abedad9175043a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:16:05.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:05 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1014418853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:05.699 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:05 smithi064 podman[67449]: 2024-04-23 18:16:05.455734159 +0000 UTC m=+0.818288314 container remove b5a8e1bcf9a096516292928af5c02c0311fa30fd1db5da8c3abedad9175043a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:16:05.699 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-23T18:16:05.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:05 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1014418853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:05.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:05 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1014418853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:06.172 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-23T18:16:06.172 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.407s CPU time. 2024-04-23T18:16:06.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:06 smithi032 ceph-mon[29221]: pgmap v32: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 53 op/s 2024-04-23T18:16:06.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:06 smithi040 ceph-mon[32626]: pgmap v32: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 53 op/s 2024-04-23T18:16:06.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:06 smithi064 ceph-mon[33062]: pgmap v32: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 53 op/s 2024-04-23T18:16:08.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:08 smithi032 ceph-mon[29221]: pgmap v33: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-04-23T18:16:08.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 40.0 s, 299.7 tps, lat 2170.721 ms stddev 7896.472 2024-04-23T18:16:08.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:08 smithi040 ceph-mon[32626]: pgmap v33: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-04-23T18:16:08.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:08 smithi064 ceph-mon[33062]: pgmap v33: 97 pgs: 97 active+clean; 860 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-04-23T18:16:10.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:10 smithi032 ceph-mon[29221]: pgmap v34: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 40 MiB/s wr, 128 op/s 2024-04-23T18:16:10.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:10 smithi040 ceph-mon[32626]: pgmap v34: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 40 MiB/s wr, 128 op/s 2024-04-23T18:16:10.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:10 smithi064 ceph-mon[33062]: pgmap v34: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 40 MiB/s wr, 128 op/s 2024-04-23T18:16:12.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:12 smithi032 ceph-mon[29221]: pgmap v35: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 105 op/s 2024-04-23T18:16:12.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:12 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:12.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:12 smithi040 ceph-mon[32626]: pgmap v35: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 105 op/s 2024-04-23T18:16:12.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:12 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:12.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:12 smithi064 ceph-mon[33062]: pgmap v35: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 105 op/s 2024-04-23T18:16:12.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:12 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:13.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 45.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:14.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:14.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:14.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:14.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:14.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:14.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:16:15.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:14 smithi040 ceph-mon[32626]: pgmap v36: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 25 MiB/s wr, 106 op/s 2024-04-23T18:16:15.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:14 smithi064 ceph-mon[33062]: pgmap v36: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 25 MiB/s wr, 106 op/s 2024-04-23T18:16:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:14 smithi032 ceph-mon[29221]: pgmap v36: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 25 MiB/s wr, 106 op/s 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:15 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:15 smithi064 systemd[1]: Stopped Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:15 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.407s CPU time. 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:15 smithi064 systemd[1]: Starting Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 podman[67677]: 2024-04-23 18:16:16.075006236 +0000 UTC m=+0.019380692 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:16.175 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 podman[67677]: 2024-04-23 18:16:16.182229349 +0000 UTC m=+0.126603775 container create 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:16:16.835 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 podman[67677]: 2024-04-23 18:16:16.551683043 +0000 UTC m=+0.496057483 container init 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:16:16.835 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 podman[67677]: 2024-04-23 18:16:16.554737799 +0000 UTC m=+0.499112226 container start 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:16:16.835 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 ceph-mds[67696]: -- 172.21.15.64:0/1677187680 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b37ad63a40 con 0x55b37ad36c00 2024-04-23T18:16:16.835 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: starting mds.c at 2024-04-23T18:16:16.835 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 bash[67677]: 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c 2024-04-23T18:16:16.836 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:16:16 smithi064 systemd[1]: Started Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 -- 172.21.15.32:0/1291655103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 msgr2=0x7f63301025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 --2- 172.21.15.32:0/1291655103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f63301025a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6328009900 tx=0x7f632802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 -- 172.21.15.32:0/1291655103 shutdown_connections 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 --2- 172.21.15.32:0/1291655103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 0x7f633010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 --2- 172.21.15.32:0/1291655103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6330102b70 0x7f6330106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 --2- 172.21.15.32:0/1291655103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f63301025a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 -- 172.21.15.32:0/1291655103 >> 172.21.15.32:0/1291655103 conn(0x7f63300fda40 msgr2=0x7f63300ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 -- 172.21.15.32:0/1291655103 shutdown_connections 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.914+0000 7f63383c6640 1 -- 172.21.15.32:0/1291655103 wait complete. 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 Processor -- start 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 -- start start 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 0x7f633019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63301985e0 0x7f6330198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6330112240 con 0x7f63301021c0 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63301120c0 con 0x7f6330107290 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f63383c6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f63301123c0 con 0x7f63301985e0 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f6336162640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f6336162640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54900/0 (socket says 172.21.15.32:54900) 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f6336162640 1 -- 172.21.15.32:0/1575269232 learned_addr learned my addr 172.21.15.32:0/1575269232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f6335961640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 0x7f633019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.915+0000 7f6336963640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63301985e0 0x7f6330198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 -- 172.21.15.32:0/1575269232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63301985e0 msgr2=0x7f6330198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63301985e0 0x7f6330198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 -- 172.21.15.32:0/1575269232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 msgr2=0x7f633019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 0x7f633019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 -- 172.21.15.32:0/1575269232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6330199100 con 0x7f63301021c0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f6336162640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6328005ab0 tx=0x7f6328003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f632f7fe640 1 -- 172.21.15.32:0/1575269232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632803f070 con 0x7f63301021c0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f632f7fe640 1 -- 172.21.15.32:0/1575269232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6328031d60 con 0x7f63301021c0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6330199390 con 0x7f63301021c0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f632f7fe640 1 -- 172.21.15.32:0/1575269232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328035030 con 0x7f63301021c0 2024-04-23T18:16:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.916+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63301a5580 con 0x7f63301021c0 2024-04-23T18:16:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.918+0000 7f632f7fe640 1 -- 172.21.15.32:0/1575269232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7f6328042070 con 0x7f63301021c0 2024-04-23T18:16:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 Processor -- start 2024-04-23T18:16:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 -- start start 2024-04-23T18:16:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628cfe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfee00 0x558628d0d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558628c63970 con 0x558628d0dc10 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558628c004a0 con 0x558628cfe4e0 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f633b1b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558628c5f930 con 0x558628cfee00 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f632f7fe640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 0x7f63180a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f6336963640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628cfe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54902/0 (socket says 172.21.15.32:54902) 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.919+0000 7f6335961640 1 -- 172.21.15.32:0/170766217 learned_addr learned my addr 172.21.15.32:0/170766217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6336963640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 0x7f63180a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f632f7fe640 1 -- 172.21.15.32:0/1575269232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f63280f0d80 con 0x7f63301021c0 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6336162640 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfee00 0x558628d0d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 -- 172.21.15.32:0/170766217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfee00 msgr2=0x558628d0d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfee00 0x558628d0d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 -- 172.21.15.32:0/170766217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 msgr2=0x558628cfe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628cfe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 -- 172.21.15.32:0/170766217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558628976210 con 0x558628d0dc10 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6335961640 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f632400b690 tx=0x7f632400bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49285738ca763e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f6336963640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 0x7f63180a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6320005fd0 tx=0x7f6320009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f632cbfe640 1 -- 172.21.15.32:0/170766217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324003bd0 con 0x558628d0dc10 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f632cbfe640 1 -- 172.21.15.32:0/170766217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6324015070 con 0x558628d0dc10 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.920+0000 7f632cbfe640 1 -- 172.21.15.32:0/170766217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324004cf0 con 0x558628d0dc10 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 -- 172.21.15.32:0/170766217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 msgr2=0x558628d10000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f632400b690 tx=0x7f632400bb50 comp rx=0 tx=0).stop 2024-04-23T18:16:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 -- 172.21.15.32:0/170766217 shutdown_connections 2024-04-23T18:16:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d10000 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfee00 0x558628d0d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 --2- 172.21.15.32:0/170766217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628cfe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 -- 172.21.15.32:0/170766217 >> 172.21.15.32:0/170766217 conn(0x558628cfab30 msgr2=0x558628cfaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 -- 172.21.15.32:0/170766217 shutdown_connections 2024-04-23T18:16:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.921+0000 7f633b1b9740 1 -- 172.21.15.32:0/170766217 wait complete. 2024-04-23T18:16:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 -- start start 2024-04-23T18:16:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 Processor -- start 2024-04-23T18:16:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfee00 0x558628d0a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558628d107e0 con 0x558628d0dc10 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558628bfe0b0 con 0x558628cfee00 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558628d10960 con 0x558628cfe4e0 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f633b1b9740 10 client.0 _ll_register_callbacks cb 0x558628bebd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54906/0 (socket says 172.21.15.32:54906) 2024-04-23T18:16:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6335961640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfee00 0x558628d0a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:56716/0 (socket says 172.21.15.32:56716) 2024-04-23T18:16:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 -- 172.21.15.32:0/3173580506 learned_addr learned my addr 172.21.15.32:0/3173580506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 -- 172.21.15.32:0/3173580506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 msgr2=0x558628d0d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 -- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfee00 msgr2=0x558628d0a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfee00 0x558628d0a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336963640 1 -- 172.21.15.32:0/3173580506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558628a42620 con 0x558628d0dc10 2024-04-23T18:16:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.922+0000 7f6336162640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6336963640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f632000ff20 tx=0x7f6320004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c637b1844bf337f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632001e410 con 0x558628d0dc10 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f632001e5b0 con 0x558628d0dc10 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6324015070 con 0x558628d0dc10 2024-04-23T18:16:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f632001e7b0 con 0x558628d0dc10 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 10 client.15717 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558628d1c740 con 0x558628d0dc10 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6324015070 con 0x558628d0dc10 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f632001d070 con 0x558628d0dc10 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 10 client.15717 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 20 client.15717 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d1dc70 0x558628d200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 4 client.15717 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.923+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558628c71390 con 0x558628d1dc70 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.924+0000 7f6336162640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d1dc70 0x558628d200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.924+0000 7f6336162640 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d1dc70 0x558628d200a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3682f458c5444e server_cookie=e24e53f8ffa06b3f in_seq=0 out_seq=0 2024-04-23T18:16:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.924+0000 7f6315ffb640 10 client.15717 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f6315ffb640 1 -- 172.21.15.32:0/3173580506 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558628c71390 con 0x558628d1dc70 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f6315ffb640 10 client.15717 handle_command_reply: tid=0 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 client.15717 shutdown 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 msgr2=0x558628d0ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f632000ff20 tx=0x7f6320004ab0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 shutdown_connections 2024-04-23T18:16:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d1dc70 0x558628d200a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628d0dc10 0x558628d0ad60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfee00 0x558628d0a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f6315ffb640 0 client.15717 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3173580506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d0d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 >> 172.21.15.32:0/3173580506 conn(0x558628cfab30 msgr2=0x558628d008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 shutdown_connections 2024-04-23T18:16:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 1 -- 172.21.15.32:0/3173580506 wait complete. 2024-04-23T18:16:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.925+0000 7f633b1b9740 20 client.15717 trim_cache size 0 max 16384 2024-04-23T18:16:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 Processor -- start 2024-04-23T18:16:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 -- start start 2024-04-23T18:16:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d196e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628d0dc10 0x558628d25ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63301ae1a0 con 0x558628cfee00 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63301aba10 con 0x558628d0dc10 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f633b1b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f63301a9880 con 0x558628cfe4e0 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54908/0 (socket says 172.21.15.32:54908) 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f6335961640 1 -- 172.21.15.32:0/3180832634 learned_addr learned my addr 172.21.15.32:0/3180832634 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f6336963640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628d0dc10 0x558628d25ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.940+0000 7f6336162640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 -- 172.21.15.32:0/3180832634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 msgr2=0x558628d196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d196e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 -- 172.21.15.32:0/3180832634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628d0dc10 msgr2=0x558628d25ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628d0dc10 0x558628d25ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 -- 172.21.15.32:0/3180832634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558628cfec30 con 0x558628cfee00 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6336162640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6335961640 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6324004920 tx=0x7f6324004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd205765d815d862 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6317fff640 1 -- 172.21.15.32:0/3180832634 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324029070 con 0x558628cfee00 2024-04-23T18:16:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6317fff640 1 -- 172.21.15.32:0/3180832634 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6324015050 con 0x558628cfee00 2024-04-23T18:16:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.941+0000 7f6317fff640 1 -- 172.21.15.32:0/3180832634 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324024db0 con 0x558628cfee00 2024-04-23T18:16:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- 172.21.15.32:0/3180832634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 msgr2=0x558628d259b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6324004920 tx=0x7f6324004580 comp rx=0 tx=0).stop 2024-04-23T18:16:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- 172.21.15.32:0/3180832634 shutdown_connections 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628d0dc10 0x558628d25ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d259b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- 172.21.15.32:0/3180832634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628cfe4e0 0x558628d196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- 172.21.15.32:0/3180832634 >> 172.21.15.32:0/3180832634 conn(0x558628cfab30 msgr2=0x558628d0a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- 172.21.15.32:0/3180832634 shutdown_connections 2024-04-23T18:16:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- 172.21.15.32:0/3180832634 wait complete. 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- start start 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 Processor -- start 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628d115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628d0dc10 0x558628d12050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63301a9ed0 con 0x558628cfee00 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63301ab340 con 0x558628cfe4e0 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f63301aa5a0 con 0x558628d0dc10 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f633b1b9740 10 client.0 _ll_register_callbacks cb 0x7f6320009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f6335961640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54910/0 (socket says 172.21.15.32:54910) 2024-04-23T18:16:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f6336162640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628d115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.942+0000 7f6335961640 1 -- 172.21.15.32:0/1467959038 learned_addr learned my addr 172.21.15.32:0/1467959038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6336963640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628d0dc10 0x558628d12050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 -- 172.21.15.32:0/1467959038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628d0dc10 msgr2=0x558628d12050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628d0dc10 0x558628d12050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 -- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 msgr2=0x558628d115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628d115d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 -- 172.21.15.32:0/1467959038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558628d83220 con 0x558628cfee00 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6336162640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628d115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6335961640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f632400bb90 tx=0x7f6324005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92178d1a4832dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324029070 con 0x558628cfee00 2024-04-23T18:16:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6324003de0 con 0x558628cfee00 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6324015050 con 0x558628cfee00 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6324024470 con 0x558628cfee00 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f633b1b9740 10 client.15723 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558628d84490 con 0x558628cfee00 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.943+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6324015050 con 0x558628cfee00 2024-04-23T18:16:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f6324019520 con 0x558628cfee00 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 10 client.15723 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 20 client.15723 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d849b0 0x558628d86da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 4 client.15723 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:16:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558628c6b200 con 0x558628d849b0 2024-04-23T18:16:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f6336162640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d849b0 0x558628d86da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f6336162640 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d849b0 0x558628d86da0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46156a0e37772088 server_cookie=1ac12d4e9606e2ef in_seq=0 out_seq=0 2024-04-23T18:16:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.944+0000 7f6315ffb640 10 client.15723 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.958+0000 7f6315ffb640 1 -- 172.21.15.32:0/1467959038 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558628c6b200 con 0x558628d849b0 2024-04-23T18:16:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.958+0000 7f6315ffb640 10 client.15723 handle_command_reply: tid=0 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.958+0000 7f633b1b9740 1 client.15723 shutdown 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 msgr2=0x558628d11b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f632400bb90 tx=0x7f6324005ec0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 shutdown_connections 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558628d849b0 0x558628d86da0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558628d0dc10 0x558628d12050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558628cfee00 0x558628d11b10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f6315ffb640 0 client.15723 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 --2- 172.21.15.32:0/1467959038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558628cfe4e0 0x558628d115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 >> 172.21.15.32:0/1467959038 conn(0x558628cfab30 msgr2=0x558628d05670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 shutdown_connections 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 1 -- 172.21.15.32:0/1467959038 wait complete. 2024-04-23T18:16:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.959+0000 7f633b1b9740 20 client.15723 trim_cache size 0 max 16384 2024-04-23T18:16:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:16:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:16:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:16:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 msgr2=0x7f63180a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 0x7f63180a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6320005fd0 tx=0x7f6320009500 comp rx=0 tx=0).stop 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 msgr2=0x7f633019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6328005ab0 tx=0x7f6328003c80 comp rx=0 tx=0).stop 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 shutdown_connections 2024-04-23T18:16:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63301985e0 0x7f6330198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6330107290 0x7f633019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63180a3e30 0x7f63180a62f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 --2- 172.21.15.32:0/1575269232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63301021c0 0x7f633019e430 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 >> 172.21.15.32:0/1575269232 conn(0x7f63300fda40 msgr2=0x7f63300ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 shutdown_connections 2024-04-23T18:16:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:16.960+0000 7f63383c6640 1 -- 172.21.15.32:0/1575269232 wait complete. 2024-04-23T18:16:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:16 smithi040 ceph-mon[32626]: pgmap v37: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 23 MiB/s wr, 88 op/s 2024-04-23T18:16:17.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:16 smithi040 ceph-mon[32626]: daemon mds.c restarted 2024-04-23T18:16:17.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:16 smithi064 ceph-mon[33062]: pgmap v37: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 23 MiB/s wr, 88 op/s 2024-04-23T18:16:17.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:16 smithi064 ceph-mon[33062]: daemon mds.c restarted 2024-04-23T18:16:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:16 smithi032 ceph-mon[29221]: pgmap v37: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 23 MiB/s wr, 88 op/s 2024-04-23T18:16:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:16 smithi032 ceph-mon[29221]: daemon mds.c restarted 2024-04-23T18:16:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:17 smithi040 ceph-mon[32626]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:16:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:17 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6836/3901397202,v1:172.21.15.64:6837/3901397202] up:boot 2024-04-23T18:16:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:17 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:16:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:18.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:17 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:17 smithi064 ceph-mon[33062]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:16:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:17 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6836/3901397202,v1:172.21.15.64:6837/3901397202] up:boot 2024-04-23T18:16:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:17 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:16:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:18.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:17 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:18.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:17 smithi032 ceph-mon[29221]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:16:18.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:17 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6836/3901397202,v1:172.21.15.64:6837/3901397202] up:boot 2024-04-23T18:16:18.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:17 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:16:18.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:18.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:17 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 50.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:18 smithi040 ceph-mon[32626]: pgmap v39: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 23 MiB/s wr, 100 op/s 2024-04-23T18:16:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:18 smithi064 ceph-mon[33062]: pgmap v39: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 23 MiB/s wr, 100 op/s 2024-04-23T18:16:19.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:18 smithi032 ceph-mon[29221]: pgmap v39: 97 pgs: 97 active+clean; 928 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 23 MiB/s wr, 100 op/s 2024-04-23T18:16:21.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:20 smithi040 ceph-mon[32626]: pgmap v40: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:20 smithi064 ceph-mon[33062]: pgmap v40: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:20 smithi032 ceph-mon[29221]: pgmap v40: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:23.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:22 smithi040 ceph-mon[32626]: pgmap v41: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:22 smithi064 ceph-mon[33062]: pgmap v41: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:22 smithi032 ceph-mon[29221]: pgmap v41: 97 pgs: 97 active+clean; 928 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-04-23T18:16:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 55.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:25.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:24 smithi040 ceph-mon[32626]: pgmap v42: 97 pgs: 97 active+clean; 937 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 944 KiB/s wr, 124 op/s 2024-04-23T18:16:25.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:24 smithi064 ceph-mon[33062]: pgmap v42: 97 pgs: 97 active+clean; 937 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 944 KiB/s wr, 124 op/s 2024-04-23T18:16:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:24 smithi032 ceph-mon[29221]: pgmap v42: 97 pgs: 97 active+clean; 937 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 944 KiB/s wr, 124 op/s 2024-04-23T18:16:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:26 smithi040 ceph-mon[32626]: pgmap v43: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.2 MiB/s wr, 154 op/s 2024-04-23T18:16:27.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:26 smithi064 ceph-mon[33062]: pgmap v43: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.2 MiB/s wr, 154 op/s 2024-04-23T18:16:27.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:26 smithi032 ceph-mon[29221]: pgmap v43: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.2 MiB/s wr, 154 op/s 2024-04-23T18:16:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 60.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:28 smithi040 ceph-mon[32626]: pgmap v44: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2024-04-23T18:16:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:28 smithi064 ceph-mon[33062]: pgmap v44: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2024-04-23T18:16:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:28 smithi032 ceph-mon[29221]: pgmap v44: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2024-04-23T18:16:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- 172.21.15.32:0/797382792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1014a0 msgr2=0x7fa8cc105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/797382792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1014a0 0x7fa8cc105290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009930 tx=0x7fa8bc02a420 comp rx=0 tx=0).stop 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- 172.21.15.32:0/797382792 shutdown_connections 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/797382792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc10d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/797382792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1014a0 0x7fa8cc105290 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/797382792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 0x7fa8cc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- 172.21.15.32:0/797382792 >> 172.21.15.32:0/797382792 conn(0x7fa8cc076200 msgr2=0x7fa8cc076610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- 172.21.15.32:0/797382792 shutdown_connections 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- 172.21.15.32:0/797382792 wait complete. 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 Processor -- start 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.061+0000 7fa8d3efd640 1 -- start start 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 0x7fa8cc1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc19a750 0x7fa8cc19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc110020 con 0x7fa8cc1059c0 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10fea0 con 0x7fa8cc100af0 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d3efd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1101a0 con 0x7fa8cc19a750 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41006/0 (socket says 172.21.15.32:41006) 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 -- 172.21.15.32:0/3555246458 learned_addr learned my addr 172.21.15.32:0/3555246458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d249a640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc19a750 0x7fa8cc19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 0x7fa8cc1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 -- 172.21.15.32:0/3555246458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc19a750 msgr2=0x7fa8cc19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc19a750 0x7fa8cc19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 -- 172.21.15.32:0/3555246458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 msgr2=0x7fa8cc1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 0x7fa8cc1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 -- 172.21.15.32:0/3555246458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc19b350 con 0x7fa8cc1059c0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.062+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc004770 tx=0x7fa8bc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.063+0000 7fa8c2ffd640 1 -- 172.21.15.32:0/3555246458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc03f030 con 0x7fa8cc1059c0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.063+0000 7fa8c2ffd640 1 -- 172.21.15.32:0/3555246458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8bc003aa0 con 0x7fa8cc1059c0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.063+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8cc19b5e0 con 0x7fa8cc1059c0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.063+0000 7fa8c2ffd640 1 -- 172.21.15.32:0/3555246458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc035070 con 0x7fa8cc1059c0 2024-04-23T18:16:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.063+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8cc1a7720 con 0x7fa8cc1059c0 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.065+0000 7fa8c2ffd640 1 -- 172.21.15.32:0/3555246458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fa8bc042030 con 0x7fa8cc1059c0 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.065+0000 7fa8d6c0f740 1 Processor -- start 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 -- start start 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbb6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a77cb1b970 con 0x55a77cbb6f40 2024-04-23T18:16:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a77cab84a0 con 0x55a77cbb6620 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a77cb17930 con 0x55a77cbc1800 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1c99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8c2ffd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 0x7fa8b80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1c99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41012/0 (socket says 172.21.15.32:41012) 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1c99640 1 -- 172.21.15.32:0/4125764621 learned_addr learned my addr 172.21.15.32:0/4125764621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1498640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d249a640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbb6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d249a640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 0x7fa8b80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1c99640 1 -- 172.21.15.32:0/4125764621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 msgr2=0x55a77cbc3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.066+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d1c99640 1 -- 172.21.15.32:0/4125764621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 msgr2=0x55a77cbb6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbb6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d1c99640 1 -- 172.21.15.32:0/4125764621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77c82e210 con 0x55a77cbb6f40 2024-04-23T18:16:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d1498640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8c2ffd640 1 -- 172.21.15.32:0/3555246458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa8bc0f0f90 con 0x7fa8cc1059c0 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800bd10 tx=0x7fa8c80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9d7b2a0f8fdf09 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8d249a640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 0x7fa8b80a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4005c40 tx=0x7fa8c400a610 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8b7fff640 1 -- 172.21.15.32:0/4125764621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8004030 con 0x55a77cbb6f40 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8b7fff640 1 -- 172.21.15.32:0/4125764621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8c80035f0 con 0x55a77cbb6f40 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.067+0000 7fa8b7fff640 1 -- 172.21.15.32:0/4125764621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8011680 con 0x55a77cbb6f40 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/4125764621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 msgr2=0x55a77cbc12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800bd10 tx=0x7fa8c80107a0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/4125764621 shutdown_connections 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc12c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/4125764621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbb6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/4125764621 >> 172.21.15.32:0/4125764621 conn(0x55a77cbb2b30 msgr2=0x55a77cbb2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/4125764621 shutdown_connections 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/4125764621 wait complete. 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 -- start start 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.068+0000 7fa8d6c0f740 1 Processor -- start 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbc1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbbe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a77cbc4430 con 0x55a77cbb6f40 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a77cab60b0 con 0x55a77cbb6620 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a77cbc45b0 con 0x55a77cbc1800 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d6c0f740 10 client.0 _ll_register_callbacks cb 0x55a77caa3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41020/0 (socket says 172.21.15.32:41020) 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 -- 172.21.15.32:0/3044045881 learned_addr learned my addr 172.21.15.32:0/3044045881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1c99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbc1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d249a640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbbe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 -- 172.21.15.32:0/3044045881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 msgr2=0x55a77cbbe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbbe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 -- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 msgr2=0x55a77cbc1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbc1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1498640 1 -- 172.21.15.32:0/3044045881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77c8fa620 con 0x55a77cbb6f40 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.069+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbc1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d1498640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc000c00 tx=0x7fa8bc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd0a7106fce6c83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc049070 con 0x55a77cbb6f40 2024-04-23T18:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa8bc03bc90 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c80035f0 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8bc0c1150 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 10 client.15735 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a77cbd0300 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8c80035f0 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fa8bc0ee8c0 con 0x55a77cbb6f40 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 10 client.15735 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 20 client.15735 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cbd17e0 0x55a77cbd3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 4 client.15735 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.070+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77cb29390 con 0x55a77cbd17e0 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.071+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cbd17e0 0x55a77cbd3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.071+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cbd17e0 0x55a77cbd3ca0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf4aea43418b657c server_cookie=403c98ac93a5afda in_seq=0 out_seq=0 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.071+0000 7fa8b57fa640 10 client.15735 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8b57fa640 1 -- 172.21.15.32:0/3044045881 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a77cb29390 con 0x55a77cbd17e0 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8b57fa640 10 client.15735 handle_command_reply: tid=0 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 client.15735 shutdown 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 msgr2=0x55a77cbbe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc000c00 tx=0x7fa8bc02a420 comp rx=0 tx=0).stop 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 shutdown_connections 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cbd17e0 0x55a77cbd3ca0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbbe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbbe3f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/3044045881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbc1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8b57fa640 0 client.15735 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 >> 172.21.15.32:0/3044045881 conn(0x55a77cbb2b30 msgr2=0x55a77cbb88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 shutdown_connections 2024-04-23T18:16:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/3044045881 wait complete. 2024-04-23T18:16:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.072+0000 7fa8d6c0f740 20 client.15735 trim_cache size 0 max 16384 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.087+0000 7fa8d6c0f740 1 Processor -- start 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.087+0000 7fa8d6c0f740 1 -- start start 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbcd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1b2490 con 0x55a77cbb6f40 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1afd00 con 0x55a77cbb6620 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1adb50 con 0x55a77cbc1800 2024-04-23T18:16:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41032/0 (socket says 172.21.15.32:41032) 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 -- 172.21.15.32:0/2385842015 learned_addr learned my addr 172.21.15.32:0/2385842015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbcd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d249a640 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 -- 172.21.15.32:0/2385842015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 msgr2=0x55a77cbc0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 -- 172.21.15.32:0/2385842015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 msgr2=0x55a77cbcd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbcd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 -- 172.21.15.32:0/2385842015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77cbb3700 con 0x55a77cbb6f40 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8d1498640 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0bcde0 tx=0x7fa8bc0b90f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874869df291e0dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.088+0000 7fa8b77fe640 1 -- 172.21.15.32:0/2385842015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc049070 con 0x55a77cbb6f40 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8b77fe640 1 -- 172.21.15.32:0/2385842015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8bc0c5a10 con 0x55a77cbb6f40 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8b77fe640 1 -- 172.21.15.32:0/2385842015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc0c57e0 con 0x55a77cbb6f40 2024-04-23T18:16:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/2385842015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 msgr2=0x55a77cbb9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0bcde0 tx=0x7fa8bc0b90f0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/2385842015 shutdown_connections 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbc1800 0x55a77cbc0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbb9970 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/2385842015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbb6620 0x55a77cbcd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/2385842015 >> 172.21.15.32:0/2385842015 conn(0x55a77cbb2b30 msgr2=0x55a77cbb9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/2385842015 shutdown_connections 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/2385842015 wait complete. 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.089+0000 7fa8d6c0f740 1 -- start start 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 Processor -- start 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbb6620 0x55a77cbc97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 0x55a77cbc4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1ae1c0 con 0x55a77cbb6f40 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1af630 con 0x55a77cbc1800 2024-04-23T18:16:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc1ae890 con 0x55a77cbb6620 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d6c0f740 10 client.0 _ll_register_callbacks cb 0x7fa8bc0bccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41034/0 (socket says 172.21.15.32:41034) 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 -- 172.21.15.32:0/394506343 learned_addr learned my addr 172.21.15.32:0/394506343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d249a640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 0x55a77cbc4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbb6620 0x55a77cbc97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 -- 172.21.15.32:0/394506343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbb6620 msgr2=0x55a77cbc97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbb6620 0x55a77cbc97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 -- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 msgr2=0x55a77cbc4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 0x55a77cbc4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d1498640 1 -- 172.21.15.32:0/394506343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77cbc54b0 con 0x55a77cbb6f40 2024-04-23T18:16:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.090+0000 7fa8d249a640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 0x55a77cbc4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d1498640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc031440 tx=0x7fa8bc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9abafb6068ebed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc049050 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa8bc0c5780 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8bc0c5a10 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8bc0bc7b0 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 10 client.15741 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a77cbc54b0 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8bc0c5a10 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fa8bc0dd1d0 con 0x55a77cbb6f40 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 10 client.15741 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 20 client.15741 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cc37db0 0x55a77cc3a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 4 client.15741 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.091+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a77cbbecc0 con 0x55a77cc37db0 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.092+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cc37db0 0x55a77cc3a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.092+0000 7fa8b57fa640 10 client.15741 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.092+0000 7fa8d1c99640 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cc37db0 0x55a77cc3a1d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1db23d0e4ac471ce server_cookie=2f39c9cd38f8efff in_seq=0 out_seq=0 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8b57fa640 1 -- 172.21.15.32:0/394506343 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a77cbbecc0 con 0x55a77cc37db0 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8b57fa640 10 client.15741 handle_command_reply: tid=0 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 client.15741 shutdown 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 msgr2=0x55a77cbc9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc031440 tx=0x7fa8bc02a420 comp rx=0 tx=0).stop 2024-04-23T18:16:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 shutdown_connections 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a77cc37db0 0x55a77cc3a1d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a77cbc1800 0x55a77cbc4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a77cbb6f40 0x55a77cbc9ce0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 --2- 172.21.15.32:0/394506343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a77cbb6620 0x55a77cbc97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8b57fa640 0 client.15741 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 >> 172.21.15.32:0/394506343 conn(0x55a77cbb2b30 msgr2=0x55a77cbb1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 shutdown_connections 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 1 -- 172.21.15.32:0/394506343 wait complete. 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.107+0000 7fa8d6c0f740 20 client.15741 trim_cache size 0 max 16384 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:16:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:16:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:16:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:16:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 msgr2=0x7fa8b80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 0x7fa8b80a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4005c40 tx=0x7fa8c400a610 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 msgr2=0x7fa8cc1a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc004770 tx=0x7fa8bc004270 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 shutdown_connections 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8cc19a750 0x7fa8cc19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8b80a3e30 0x7fa8b80a62f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc1059c0 0x7fa8cc1a0bd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 --2- 172.21.15.32:0/3555246458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8cc100af0 0x7fa8cc1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 >> 172.21.15.32:0/3555246458 conn(0x7fa8cc076200 msgr2=0x7fa8cc0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 shutdown_connections 2024-04-23T18:16:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:30.108+0000 7fa8d3efd640 1 -- 172.21.15.32:0/3555246458 wait complete. 2024-04-23T18:16:31.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:30 smithi040 ceph-mon[32626]: pgmap v45: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 994 KiB/s wr, 131 op/s 2024-04-23T18:16:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:30 smithi064 ceph-mon[33062]: pgmap v45: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 994 KiB/s wr, 131 op/s 2024-04-23T18:16:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:30 smithi032 ceph-mon[29221]: pgmap v45: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 994 KiB/s wr, 131 op/s 2024-04-23T18:16:33.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:32 smithi064 ceph-mon[33062]: pgmap v46: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 986 KiB/s wr, 124 op/s 2024-04-23T18:16:33.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:32 smithi032 ceph-mon[29221]: pgmap v46: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 986 KiB/s wr, 124 op/s 2024-04-23T18:16:33.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:32 smithi040 ceph-mon[32626]: pgmap v46: 97 pgs: 97 active+clean; 940 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 986 KiB/s wr, 124 op/s 2024-04-23T18:16:33.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 65.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:16:34.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:35.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 -- 172.21.15.32:0/4122778308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24010c590 msgr2=0x7fa24010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 --2- 172.21.15.32:0/4122778308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24010c590 0x7fa24010e9c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa238009900 tx=0x7fa23802a480 comp rx=0 tx=0).stop 2024-04-23T18:16:35.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 -- 172.21.15.32:0/4122778308 shutdown_connections 2024-04-23T18:16:35.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 --2- 172.21.15.32:0/4122778308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24010c590 0x7fa24010e9c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 --2- 172.21.15.32:0/4122778308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa240103ae0 0x7fa24010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 --2- 172.21.15.32:0/4122778308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2401031c0 0x7fa2401035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 -- 172.21.15.32:0/4122778308 >> 172.21.15.32:0/4122778308 conn(0x7fa240077f50 msgr2=0x7fa240078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 -- 172.21.15.32:0/4122778308 shutdown_connections 2024-04-23T18:16:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 -- 172.21.15.32:0/4122778308 wait complete. 2024-04-23T18:16:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.016+0000 7fa2459bf640 1 Processor -- start 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 -- start start 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2401031c0 0x7fa240100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24010c590 0x7fa240101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401121b0 con 0x7fa240103ae0 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa240112030 con 0x7fa24010c590 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa2459bf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa240112330 con 0x7fa2401031c0 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41048/0 (socket says 172.21.15.32:41048) 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 -- 172.21.15.32:0/903992051 learned_addr learned my addr 172.21.15.32:0/903992051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23f7fe640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24010c590 0x7fa240101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23effd640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2401031c0 0x7fa240100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 -- 172.21.15.32:0/903992051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2401031c0 msgr2=0x7fa240100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2401031c0 0x7fa240100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 -- 172.21.15.32:0/903992051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24010c590 msgr2=0x7fa240101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24010c590 0x7fa240101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.017+0000 7fa23e7fc640 1 -- 172.21.15.32:0/903992051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2401a6c50 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa23e7fc640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa23400e9f0 tx=0x7fa23400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23400cd80 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2340054d0 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2401a6f40 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234010690 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.018+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401a73b0 con 0x7fa240103ae0 2024-04-23T18:16:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.019+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa240069250 con 0x7fa240103ae0 2024-04-23T18:16:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.020+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fa234004e30 con 0x7fa240103ae0 2024-04-23T18:16:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.021+0000 7fa2449bd640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 0x7fa2280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.021+0000 7fa23effd640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 0x7fa2280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.021+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa2340d10b0 con 0x7fa240103ae0 2024-04-23T18:16:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.021+0000 7fa23effd640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 0x7fa2280a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa230005fd0 tx=0x7fa2300098b0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.025+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa234053ff0 con 0x7fa240103ae0 2024-04-23T18:16:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:34 smithi064 ceph-mon[33062]: pgmap v47: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-23T18:16:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.253+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa240102340 con 0x7fa240103ae0 2024-04-23T18:16:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.253+0000 7fa2449bd640 1 -- 172.21.15.32:0/903992051 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11854 (secure 0 0 0) 0x7fa23409d4f0 con 0x7fa240103ae0 2024-04-23T18:16:35.248 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:16:35.248 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:16:16.850144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:16:35.248 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-04-23T18:16:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 msgr2=0x7fa2280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 0x7fa2280a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa230005fd0 tx=0x7fa2300098b0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 msgr2=0x7fa240101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa23400e9f0 tx=0x7fa23400edf0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 shutdown_connections 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24010c590 0x7fa240101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa2280a3e40 0x7fa2280a6300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa240103ae0 0x7fa240101350 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.255+0000 7fa2459bf640 1 --2- 172.21.15.32:0/903992051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2401031c0 0x7fa240100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.256+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 >> 172.21.15.32:0/903992051 conn(0x7fa240077f50 msgr2=0x7fa2400ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.256+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 shutdown_connections 2024-04-23T18:16:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.256+0000 7fa2459bf640 1 -- 172.21.15.32:0/903992051 wait complete. 2024-04-23T18:16:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:34 smithi032 ceph-mon[29221]: pgmap v47: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-23T18:16:35.260 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 61826c57 --release --await' 2024-04-23T18:16:35.261 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 61826c57 --release --await 2024-04-23T18:16:35.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:34 smithi040 ceph-mon[32626]: pgmap v47: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 -- 172.21.15.32:0/442100055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 msgr2=0x7fedb4102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 --2- 172.21.15.32:0/442100055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb4102ec0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009930 tx=0x7fed9c02a420 comp rx=0 tx=0).stop 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 -- 172.21.15.32:0/442100055 shutdown_connections 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 --2- 172.21.15.32:0/442100055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 0x7fedb410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 --2- 172.21.15.32:0/442100055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb4102ec0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 --2- 172.21.15.32:0/442100055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb4101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 -- 172.21.15.32:0/442100055 >> 172.21.15.32:0/442100055 conn(0x7fedb4077f70 msgr2=0x7fedb4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 -- 172.21.15.32:0/442100055 shutdown_connections 2024-04-23T18:16:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.376+0000 7fedb882a640 1 -- 172.21.15.32:0/442100055 wait complete. 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 Processor -- start 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 -- start start 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 0x7fedb4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41122c0 con 0x7fedb4102a60 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4112140 con 0x7fedb4101180 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb882a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4112440 con 0x7fedb4103400 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41064/0 (socket says 172.21.15.32:41064) 2024-04-23T18:16:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb1d9b640 1 -- 172.21.15.32:0/3228476993 learned_addr learned my addr 172.21.15.32:0/3228476993 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.377+0000 7fedb259c640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 0x7fedb4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 -- 172.21.15.32:0/3228476993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 msgr2=0x7fedb4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 0x7fedb4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 -- 172.21.15.32:0/3228476993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 msgr2=0x7fedb419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 -- 172.21.15.32:0/3228476993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb4198fc0 con 0x7fedb4102a60 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb259c640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005940 tx=0x7fed9c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedab7fe640 1 -- 172.21.15.32:0/3228476993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c03f030 con 0x7fedb4102a60 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedab7fe640 1 -- 172.21.15.32:0/3228476993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed9c003cb0 con 0x7fedb4102a60 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb4199250 con 0x7fedb4102a60 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedab7fe640 1 -- 172.21.15.32:0/3228476993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c035070 con 0x7fedb4102a60 2024-04-23T18:16:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.378+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41a5390 con 0x7fedb4102a60 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 Processor -- start 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedab7fe640 1 -- 172.21.15.32:0/3228476993 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138879+0+0 (secure 0 0 0) 0x7fed9c042030 con 0x7fedb4102a60 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 -- start start 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fb370 0x560faf5fb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60a890 0x560faf60cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560faf5606d0 con 0x560faf5fbc90 2024-04-23T18:16:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560faf4fd1a0 con 0x560faf60a890 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.381+0000 7fedbb61c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560faf55c690 con 0x560faf5fb370 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41080/0 (socket says 172.21.15.32:41080) 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 -- 172.21.15.32:0/1278621704 learned_addr learned my addr 172.21.15.32:0/1278621704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fb370 0x560faf5fb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60a890 0x560faf60cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedab7fe640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 0x7feda00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 -- 172.21.15.32:0/1278621704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fb370 msgr2=0x560faf5fb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fb370 0x560faf5fb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 -- 172.21.15.32:0/1278621704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60a890 msgr2=0x560faf60cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60a890 0x560faf60cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 -- 172.21.15.32:0/1278621704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560faf5075b0 con 0x560faf5fbc90 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 0x7feda00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedab7fe640 1 -- 172.21.15.32:0/3228476993 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fed9c0f10c0 con 0x7fedb4102a60 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7fedb259c640 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feda4011540 tx=0x7feda4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82a48b47f70bf7c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7feda8ff9640 1 -- 172.21.15.32:0/1278621704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda400f640 con 0x560faf5fbc90 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7feda8ff9640 1 -- 172.21.15.32:0/1278621704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda400fc90 con 0x560faf5fbc90 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.382+0000 7feda8ff9640 1 -- 172.21.15.32:0/1278621704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4019cc0 con 0x560faf5fbc90 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- 172.21.15.32:0/1278621704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 msgr2=0x560faf60a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 0x7feda00a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fedac005fd0 tx=0x7fedac004a40 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feda4011540 tx=0x7feda4011a00 comp rx=0 tx=0).stop 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- 172.21.15.32:0/1278621704 shutdown_connections 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60a890 0x560faf60cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf60a350 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1278621704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fb370 0x560faf5fb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- 172.21.15.32:0/1278621704 >> 172.21.15.32:0/1278621704 conn(0x560faf5f77b0 msgr2=0x560faf5f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- 172.21.15.32:0/1278621704 shutdown_connections 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- 172.21.15.32:0/1278621704 wait complete. 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.383+0000 7fedbb61c740 1 -- start start 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 Processor -- start 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fbc90 0x560faf607530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf607ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560faf60d770 con 0x560faf5fb370 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560faf4fab20 con 0x560faf5fbc90 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560faf60d8f0 con 0x560faf60a890 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedbb61c740 10 client.0 _ll_register_callbacks cb 0x560faf4fa530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41096/0 (socket says 172.21.15.32:41096) 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 -- 172.21.15.32:0/1304914122 learned_addr learned my addr 172.21.15.32:0/1304914122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fbc90 0x560faf607530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf607ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 -- 172.21.15.32:0/1304914122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 msgr2=0x560faf607ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf607ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 -- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fbc90 msgr2=0x560faf607530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fbc90 0x560faf607530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 -- 172.21.15.32:0/1304914122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560faf272920 con 0x560faf5fb370 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fedb259c640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feda40097b0 tx=0x7feda4019bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd54c1ab199d8d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.384+0000 7fed99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda400fe40 con 0x560faf5fb370 2024-04-23T18:16:35.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7feda400f640 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda400fc90 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feda4030050 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 10 client.15759 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560faf33ed30 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda400fc90 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7feda40049e0 con 0x560faf5fb370 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 10 client.15759 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 10 client.15759 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 20 client.15759 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf61aa20 0x560faf61ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 4 client.15759 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560faf51e500 con 0x560faf61aa20 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.385+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf61aa20 0x560faf61ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.386+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf61aa20 0x560faf61ce10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=596028e18ab897f5 server_cookie=81839358c4713179 in_seq=0 out_seq=0 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.386+0000 7fed99ffb640 10 client.15759 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fed99ffb640 1 -- 172.21.15.32:0/1304914122 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560faf51e500 con 0x560faf61aa20 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fed99ffb640 10 client.15759 handle_command_reply: tid=0 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 client.15759 shutdown 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 msgr2=0x560faf60a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feda40097b0 tx=0x7feda4019bf0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 shutdown_connections 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf61aa20 0x560faf61ce10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf607ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fbc90 0x560faf607530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fed99ffb640 0 client.15759 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 --2- 172.21.15.32:0/1304914122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fb370 0x560faf60a240 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 >> 172.21.15.32:0/1304914122 conn(0x560faf5f77b0 msgr2=0x560faf5fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 shutdown_connections 2024-04-23T18:16:35.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 1 -- 172.21.15.32:0/1304914122 wait complete. 2024-04-23T18:16:35.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.387+0000 7fedbb61c740 20 client.15759 trim_cache size 0 max 16384 2024-04-23T18:16:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.404+0000 7fedbb61c740 1 Processor -- start 2024-04-23T18:16:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.404+0000 7fedbb61c740 1 -- start start 2024-04-23T18:16:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf6166a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf622610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ae220 con 0x560faf5fbc90 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41aba90 con 0x560faf5fb370 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedbb61c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41a9900 con 0x560faf60a890 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41112/0 (socket says 172.21.15.32:41112) 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 -- 172.21.15.32:0/599148695 learned_addr learned my addr 172.21.15.32:0/599148695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb259c640 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf6166a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf622610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 -- 172.21.15.32:0/599148695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 msgr2=0x560faf622610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf622610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 -- 172.21.15.32:0/599148695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 msgr2=0x560faf6166a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf6166a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 -- 172.21.15.32:0/599148695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560faf607e40 con 0x560faf5fbc90 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0bcf60 tx=0x7fed9c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d87ef0478b2cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fed9bfff640 1 -- 172.21.15.32:0/599148695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c049070 con 0x560faf5fbc90 2024-04-23T18:16:35.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fed9bfff640 1 -- 172.21.15.32:0/599148695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed9c03b460 con 0x560faf5fbc90 2024-04-23T18:16:35.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.405+0000 7fed9bfff640 1 -- 172.21.15.32:0/599148695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0c5c40 con 0x560faf5fbc90 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- 172.21.15.32:0/599148695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 msgr2=0x560faf609d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0bcf60 tx=0x7fed9c02a420 comp rx=0 tx=0).stop 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- 172.21.15.32:0/599148695 shutdown_connections 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf60a890 0x560faf622610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf5fbc90 0x560faf609d70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 --2- 172.21.15.32:0/599148695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf6166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- 172.21.15.32:0/599148695 >> 172.21.15.32:0/599148695 conn(0x560faf5f77b0 msgr2=0x560faf605e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- 172.21.15.32:0/599148695 shutdown_connections 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- 172.21.15.32:0/599148695 wait complete. 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 -- start start 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.406+0000 7fedbb61c740 1 Processor -- start 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf60b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fbc90 0x560faf60b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41a9f50 con 0x560faf60a890 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ab3c0 con 0x560faf5fb370 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41aa620 con 0x560faf5fbc90 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 10 client.0 _ll_register_callbacks cb 0x7feda4019790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41120/0 (socket says 172.21.15.32:41120) 2024-04-23T18:16:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 -- 172.21.15.32:0/2088649850 learned_addr learned my addr 172.21.15.32:0/2088649850 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb259c640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf60b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb1d9b640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fbc90 0x560faf60b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 -- 172.21.15.32:0/2088649850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fbc90 msgr2=0x560faf60b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fbc90 0x560faf60b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 -- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 msgr2=0x560faf60b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf60b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 -- 172.21.15.32:0/2088649850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560faf61d180 con 0x560faf60a890 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedb2d9d640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fedac0116e0 tx=0x7fedac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30b9f38d3d94863 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fed99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac0138f0 con 0x560faf60a890 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fedac013a90 con 0x560faf60a890 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.407+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560faf56e0f0 con 0x560faf60a890 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fedac02f020 con 0x560faf60a890 2024-04-23T18:16:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 10 client.15765 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560faf613160 con 0x560faf60a890 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560faf56e0f0 con 0x560faf60a890 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fedac02b420 con 0x560faf60a890 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 10 client.15765 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 10 client.15765 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 20 client.15765 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf6811b0 0x560faf6835d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 4 client.15765 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "61826c57", "release": true, "await": true}] 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "61826c57", "release": true, "await": true}) -- 0x560faf567f60 con 0x560faf6811b0 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.408+0000 7fedb259c640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf6811b0 0x560faf6835d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.409+0000 7fed99ffb640 10 client.15765 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.409+0000 7fedb259c640 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf6811b0 0x560faf6835d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf110600f108c1f server_cookie=fcc3cd05c2872b25 in_seq=0 out_seq=0 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.428+0000 7fed99ffb640 1 -- 172.21.15.32:0/2088649850 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x560faf567f60 con 0x560faf6811b0 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.428+0000 7fed99ffb640 10 client.15765 handle_command_reply: tid=0 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.428+0000 7fedbb61c740 1 client.15765 shutdown 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.428+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 msgr2=0x560faf611bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.428+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fedac0116e0 tx=0x7fedac004ab0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 shutdown_connections 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560faf6811b0 0x560faf6835d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560faf60a890 0x560faf611bb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560faf5fbc90 0x560faf60b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fed99ffb640 0 client.15765 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 --2- 172.21.15.32:0/2088649850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf5fb370 0x560faf60b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 >> 172.21.15.32:0/2088649850 conn(0x560faf5f77b0 msgr2=0x560faf5f69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 shutdown_connections 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 1 -- 172.21.15.32:0/2088649850 wait complete. 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedbb61c740 20 client.15765 trim_cache size 0 max 16384 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 26, 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 8, 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:16:35.424 INFO:teuthology.orchestra.run.smithi032.stdout: "61826c57": { 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 8, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:16:35.425 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:16:35.426 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 msgr2=0x7feda00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 0x7feda00a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fedac005fd0 tx=0x7fedac004a40 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 msgr2=0x7fedb419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005940 tx=0x7fed9c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 shutdown_connections 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4103400 0x7fedb4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feda00a3e40 0x7feda00a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4102a60 0x7fedb419eb30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 --2- 172.21.15.32:0/3228476993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fedb4101180 0x7fedb419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 >> 172.21.15.32:0/3228476993 conn(0x7fedb4077f70 msgr2=0x7fedb4104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 shutdown_connections 2024-04-23T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:16:35.429+0000 7fedb882a640 1 -- 172.21.15.32:0/3228476993 wait complete. 2024-04-23T18:16:35.439 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '61826c57', seconds elapsed: 31.5 2024-04-23T18:16:36.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:35 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/903992051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:35 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/903992051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:35 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/903992051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:36 smithi064 ceph-mon[33062]: pgmap v48: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-23T18:16:37.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:36 smithi032 ceph-mon[29221]: pgmap v48: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-23T18:16:37.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:36 smithi040 ceph-mon[32626]: pgmap v48: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-23T18:16:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 70.0 s, 2524.0 tps, lat 269.891 ms stddev 2719.374 2024-04-23T18:16:39.168 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:38 smithi040 ceph-mon[32626]: pgmap v49: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T18:16:39.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:38 smithi064 ceph-mon[33062]: pgmap v49: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T18:16:39.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:38 smithi032 ceph-mon[29221]: pgmap v49: 97 pgs: 97 active+clean; 940 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T18:16:41.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:40 smithi064 ceph-mon[33062]: pgmap v50: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.1 MiB/s wr, 319 op/s 2024-04-23T18:16:41.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:40 smithi064 ceph-mon[33062]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-23T18:16:41.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:40 smithi032 ceph-mon[29221]: pgmap v50: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.1 MiB/s wr, 319 op/s 2024-04-23T18:16:41.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:40 smithi032 ceph-mon[29221]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-23T18:16:41.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:40 smithi040 ceph-mon[32626]: pgmap v50: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.1 MiB/s wr, 319 op/s 2024-04-23T18:16:41.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:40 smithi040 ceph-mon[32626]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-23T18:16:43.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:42 smithi064 ceph-mon[33062]: pgmap v51: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.1 MiB/s wr, 316 op/s 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 -- 172.21.15.32:0/1034506779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad411b7f0 msgr2=0x7faad411dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1034506779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad411b7f0 0x7faad411dbe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faacc009cd0 tx=0x7faacc02a9a0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 -- 172.21.15.32:0/1034506779 shutdown_connections 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1034506779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad411b7f0 0x7faad411dbe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1034506779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406c0d0 0x7faad411b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1034506779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faad406b7b0 0x7faad406bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 -- 172.21.15.32:0/1034506779 >> 172.21.15.32:0/1034506779 conn(0x7faad406f030 msgr2=0x7faad4108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 -- 172.21.15.32:0/1034506779 shutdown_connections 2024-04-23T18:16:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.245+0000 7faad8f5d640 1 -- 172.21.15.32:0/1034506779 wait complete. 2024-04-23T18:16:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.246+0000 7faad8f5d640 1 Processor -- start 2024-04-23T18:16:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.246+0000 7faad8f5d640 1 -- start start 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 0x7faad41a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faad411b7f0 0x7faad41abff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faad4121120 con 0x7faad406b7b0 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faad4120fa0 con 0x7faad406c0d0 2024-04-23T18:16:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad8f5d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faad41212a0 con 0x7faad411b7f0 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33294/0 (socket says 172.21.15.32:33294) 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad259c640 1 -- 172.21.15.32:0/1176886301 learned_addr learned my addr 172.21.15.32:0/1176886301 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.247+0000 7faad1d9b640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 0x7faad41a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 -- 172.21.15.32:0/1176886301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faad411b7f0 msgr2=0x7faad41abff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faad411b7f0 0x7faad41abff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 -- 172.21.15.32:0/1176886301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 msgr2=0x7faad41a7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 0x7faad41a7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 -- 172.21.15.32:0/1176886301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41ac720 con 0x7faad406b7b0 2024-04-23T18:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad1d9b640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 0x7faad41a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.248+0000 7faad259c640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faac400b450 tx=0x7faac400b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.249+0000 7faac37fe640 1 -- 172.21.15.32:0/1176886301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4015020 con 0x7faad406b7b0 2024-04-23T18:16:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.249+0000 7faac37fe640 1 -- 172.21.15.32:0/1176886301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac4011ce0 con 0x7faad406b7b0 2024-04-23T18:16:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.249+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad41aca10 con 0x7faad406b7b0 2024-04-23T18:16:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.249+0000 7faac37fe640 1 -- 172.21.15.32:0/1176886301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4010de0 con 0x7faad406b7b0 2024-04-23T18:16:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.249+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41b42a0 con 0x7faad406b7b0 2024-04-23T18:16:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.253+0000 7faac37fe640 1 -- 172.21.15.32:0/1176886301 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7faac4018020 con 0x7faad406b7b0 2024-04-23T18:16:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.253+0000 7faadbc73740 1 Processor -- start 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 -- start start 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faac37fe640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 0x7faabc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f84791a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f8479c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f8479ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558f846f6970 con 0x558f84791f40 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558f846934a0 con 0x558f84791620 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faadbc73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558f846f2930 con 0x558f8479c800 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f8479c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f8479c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33310/0 (socket says 172.21.15.32:33310) 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f84791a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad1d9b640 1 -- 172.21.15.32:0/313922897 learned_addr learned my addr 172.21.15.32:0/313922897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad2d9d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 0x7faabc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad2d9d640 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f8479ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faac37fe640 1 -- 172.21.15.32:0/1176886301 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7faac40cfee0 con 0x7faad406b7b0 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.254+0000 7faad259c640 1 -- 172.21.15.32:0/313922897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 msgr2=0x558f8479ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad259c640 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f8479ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad259c640 1 -- 172.21.15.32:0/313922897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 msgr2=0x558f8479c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad259c640 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f8479c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad259c640 1 -- 172.21.15.32:0/313922897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f84409210 con 0x558f84791620 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad259c640 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f84791a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faac40cd790 tx=0x7faac4010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed57476844da537 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faac0ff9640 1 -- 172.21.15.32:0/313922897 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac40a0390 con 0x558f84791620 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faac0ff9640 1 -- 172.21.15.32:0/313922897 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac40a4c50 con 0x558f84791620 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faac0ff9640 1 -- 172.21.15.32:0/313922897 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac409b430 con 0x558f84791620 2024-04-23T18:16:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.255+0000 7faad2d9d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 0x7faabc0a6250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc02f740 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 -- 172.21.15.32:0/313922897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 msgr2=0x558f84791a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f84791a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faac40cd790 tx=0x7faac4010660 comp rx=0 tx=0).stop 2024-04-23T18:16:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 -- 172.21.15.32:0/313922897 shutdown_connections 2024-04-23T18:16:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f8479ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f8479c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 --2- 172.21.15.32:0/313922897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f84791a00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.257+0000 7faadbc73740 1 -- 172.21.15.32:0/313922897 >> 172.21.15.32:0/313922897 conn(0x558f8478db30 msgr2=0x558f8478df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- 172.21.15.32:0/313922897 shutdown_connections 2024-04-23T18:16:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- 172.21.15.32:0/313922897 wait complete. 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- start start 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 Processor -- start 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8479c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84799490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f84799a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558f8479f3d0 con 0x558f84791f40 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558f846910b0 con 0x558f84791620 2024-04-23T18:16:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558f8479f550 con 0x558f8479c800 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faadbc73740 10 client.0 _ll_register_callbacks cb 0x558f8467ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.258+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84799490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84799490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33324/0 (socket says 172.21.15.32:33324) 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad1d9b640 1 -- 172.21.15.32:0/474320267 learned_addr learned my addr 172.21.15.32:0/474320267 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8479c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad2d9d640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f84799a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 -- 172.21.15.32:0/474320267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 msgr2=0x558f84799a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f84799a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 -- 172.21.15.32:0/474320267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 msgr2=0x558f84799490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84799490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 -- 172.21.15.32:0/474320267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f844d5620 con 0x558f84791620 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faad259c640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8479c1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faac4098440 tx=0x7faac40cdb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5607855bb76bff70 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faab9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:16:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4098650 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faac40a4c50 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7faac40b5040 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faac40987f0 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faadbc73740 10 client.25421 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.259+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558f8479f3d0 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac40a4c50 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7faac40b5240 con 0x558f84791620 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 10 client.25421 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 10 client.25421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 20 client.25421 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f847ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 4 client.25421 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f84704390 con 0x558f847ac4c0 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faad1d9b640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f847ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faab9ffb640 10 client.25421 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.260+0000 7faad1d9b640 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f847ae8b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16364ff8b69fe16a server_cookie=71f297c913cd783a in_seq=0 out_seq=0 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.262+0000 7faab9ffb640 1 -- 172.21.15.32:0/474320267 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f84704390 con 0x558f847ac4c0 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.262+0000 7faab9ffb640 10 client.25421 handle_command_reply: tid=0 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.262+0000 7faadbc73740 1 client.25421 shutdown 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.262+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 msgr2=0x558f8479c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8479c1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faac4098440 tx=0x7faac40cdb60 comp rx=0 tx=0).stop 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 shutdown_connections 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f847ae8b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f8479c800 0x558f84799a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faab9ffb640 0 client.25421 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84799490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 --2- 172.21.15.32:0/474320267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8479c1a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 >> 172.21.15.32:0/474320267 conn(0x558f8478db30 msgr2=0x558f847938b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 shutdown_connections 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 1 -- 172.21.15.32:0/474320267 wait complete. 2024-04-23T18:16:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.263+0000 7faadbc73740 20 client.25421 trim_cache size 0 max 16384 2024-04-23T18:16:43.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:42 smithi032 ceph-mon[29221]: pgmap v51: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.1 MiB/s wr, 316 op/s 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 Processor -- start 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 -- start start 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8478d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791f40 0x558f8479bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faad41bf370 con 0x558f8479c800 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faad41bcbe0 con 0x558f84791620 2024-04-23T18:16:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faadbc73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faad41baa30 con 0x558f84791f40 2024-04-23T18:16:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.294+0000 7faad2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8478d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33328/0 (socket says 172.21.15.32:33328) 2024-04-23T18:16:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 -- 172.21.15.32:0/499629300 learned_addr learned my addr 172.21.15.32:0/499629300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 -- 172.21.15.32:0/499629300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791f40 msgr2=0x558f8479bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791f40 0x558f8479bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 -- 172.21.15.32:0/499629300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 msgr2=0x558f8478d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8478d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 -- 172.21.15.32:0/499629300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f84793f50 con 0x558f8479c800 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.295+0000 7faad2d9d640 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faad406b5d0 tx=0x7faacc046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49ff5d11109c7e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.296+0000 7faabbfff640 1 -- 172.21.15.32:0/499629300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc034040 con 0x558f8479c800 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.296+0000 7faabbfff640 1 -- 172.21.15.32:0/499629300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc04f070 con 0x558f8479c800 2024-04-23T18:16:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.296+0000 7faabbfff640 1 -- 172.21.15.32:0/499629300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc004630 con 0x558f8479c800 2024-04-23T18:16:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.297+0000 7faadbc73740 1 -- 172.21.15.32:0/499629300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 msgr2=0x558f84794850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.297+0000 7faadbc73740 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faad406b5d0 tx=0x7faacc046000 comp rx=0 tx=0).stop 2024-04-23T18:16:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.297+0000 7faadbc73740 1 -- 172.21.15.32:0/499629300 shutdown_connections 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.298+0000 7faadbc73740 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f8479c800 0x558f84794850 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.298+0000 7faadbc73740 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791f40 0x558f8479bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.298+0000 7faadbc73740 1 --2- 172.21.15.32:0/499629300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f84791620 0x558f8478d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.298+0000 7faadbc73740 1 -- 172.21.15.32:0/499629300 >> 172.21.15.32:0/499629300 conn(0x558f8478db30 msgr2=0x558f84794140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.298+0000 7faadbc73740 1 -- 172.21.15.32:0/499629300 shutdown_connections 2024-04-23T18:16:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.299+0000 7faadbc73740 1 -- 172.21.15.32:0/499629300 wait complete. 2024-04-23T18:16:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.299+0000 7faadbc73740 1 -- start start 2024-04-23T18:16:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.299+0000 7faadbc73740 1 Processor -- start 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791620 0x558f847a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f8479c800 0x558f84796340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33342/0 (socket says 172.21.15.32:33342) 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 -- 172.21.15.32:0/867864760 learned_addr learned my addr 172.21.15.32:0/867864760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faad41bb0a0 con 0x558f84791f40 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faad41bc510 con 0x558f8479c800 2024-04-23T18:16:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faad41bb770 con 0x558f84791620 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faadbc73740 10 client.0 _ll_register_callbacks cb 0x7faac409be30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad259c640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791620 0x558f847a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 -- 172.21.15.32:0/867864760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791620 msgr2=0x558f847a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791620 0x558f847a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 -- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f8479c800 msgr2=0x558f84796340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f8479c800 0x558f84796340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.300+0000 7faad1d9b640 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f84796a80 con 0x558f84791f40 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.301+0000 7faad1d9b640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x558f84798cf0 tx=0x7faac8005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf6dbaeb1c66cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.301+0000 7faab9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8014d10 con 0x558f84791f40 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7faac801f420 con 0x558f84791f40 2024-04-23T18:16:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faacc04f070 con 0x558f84791f40 2024-04-23T18:16:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faac802e050 con 0x558f84791f40 2024-04-23T18:16:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faadbc73740 10 client.15780 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558f84812850 con 0x558f84791f40 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.302+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc04f070 con 0x558f84791f40 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7faac8015420 con 0x558f84791f40 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 10 client.15780 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 20 client.15780 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f84815020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 4 client.15780 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:16:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f846fe200 con 0x558f847ac4c0 2024-04-23T18:16:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.303+0000 7faad259c640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f84815020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.304+0000 7faad259c640 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f84815020 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aa39f07adaed65b server_cookie=fd315a057158abf0 in_seq=0 out_seq=0 2024-04-23T18:16:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.304+0000 7faab9ffb640 10 client.15780 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faab9ffb640 1 -- 172.21.15.32:0/867864760 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f846fe200 con 0x558f847ac4c0 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faab9ffb640 10 client.15780 handle_command_reply: tid=0 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 client.15780 shutdown 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 msgr2=0x558f84795e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x558f84798cf0 tx=0x7faac8005d90 comp rx=0 tx=0).stop 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 shutdown_connections 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558f847ac4c0 0x558f84815020 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f8479c800 0x558f84796340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faab9ffb640 0 client.15780 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558f84791f40 0x558f84795e00 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 --2- 172.21.15.32:0/867864760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f84791620 0x558f847a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 >> 172.21.15.32:0/867864760 conn(0x558f8478db30 msgr2=0x558f847937d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.320+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 shutdown_connections 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faadbc73740 1 -- 172.21.15.32:0/867864760 wait complete. 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faadbc73740 20 client.15780 trim_cache size 0 max 16384 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:16:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 msgr2=0x7faabc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 0x7faabc0a6250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc02f740 comp rx=0 tx=0).stop 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 msgr2=0x7faad41a7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faac400b450 tx=0x7faac400b850 comp rx=0 tx=0).stop 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 shutdown_connections 2024-04-23T18:16:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faad411b7f0 0x7faad41abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad406c0d0 0x7faad41a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faabc0a3d90 0x7faabc0a6250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 --2- 172.21.15.32:0/1176886301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad406b7b0 0x7faad41a7780 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 >> 172.21.15.32:0/1176886301 conn(0x7faad406f030 msgr2=0x7faad41192a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 shutdown_connections 2024-04-23T18:16:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:43.321+0000 7faad8f5d640 1 -- 172.21.15.32:0/1176886301 wait complete. 2024-04-23T18:16:43.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:42 smithi040 ceph-mon[32626]: pgmap v51: 97 pgs: 97 active+clean; 970 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.1 MiB/s wr, 316 op/s 2024-04-23T18:16:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 75.0 s, 3818.2 tps, lat 26.182 ms stddev 27.653 2024-04-23T18:16:45.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:44 smithi032 ceph-mon[29221]: pgmap v52: 97 pgs: 97 active+clean; 1004 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.5 MiB/s wr, 615 op/s 2024-04-23T18:16:45.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:44 smithi064 ceph-mon[33062]: pgmap v52: 97 pgs: 97 active+clean; 1004 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.5 MiB/s wr, 615 op/s 2024-04-23T18:16:45.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:44 smithi040 ceph-mon[32626]: pgmap v52: 97 pgs: 97 active+clean; 1004 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.5 MiB/s wr, 615 op/s 2024-04-23T18:16:47.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:46 smithi064 ceph-mon[33062]: pgmap v53: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:47.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:46 smithi032 ceph-mon[29221]: pgmap v53: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:46 smithi040 ceph-mon[32626]: pgmap v53: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 80.0 s, 3562.0 tps, lat 28.141 ms stddev 29.676 2024-04-23T18:16:49.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:48 smithi064 ceph-mon[33062]: pgmap v54: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:49.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:48 smithi032 ceph-mon[29221]: pgmap v54: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:49.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:48 smithi040 ceph-mon[32626]: pgmap v54: 97 pgs: 97 active+clean; 1006 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.8 MiB/s wr, 638 op/s 2024-04-23T18:16:51.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:50 smithi064 ceph-mon[33062]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 949 op/s 2024-04-23T18:16:51.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:50 smithi032 ceph-mon[29221]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 949 op/s 2024-04-23T18:16:51.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:50 smithi040 ceph-mon[32626]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 949 op/s 2024-04-23T18:16:53.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:52 smithi064 ceph-mon[33062]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 641 op/s 2024-04-23T18:16:53.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:52 smithi032 ceph-mon[29221]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 641 op/s 2024-04-23T18:16:53.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:52 smithi040 ceph-mon[32626]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 641 op/s 2024-04-23T18:16:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 85.0 s, 3994.0 tps, lat 25.027 ms stddev 24.246 2024-04-23T18:16:55.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:54 smithi064 ceph-mon[33062]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-23T18:16:55.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:54 smithi032 ceph-mon[29221]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-23T18:16:55.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:54 smithi040 ceph-mon[32626]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/2519428336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b6410c590 msgr2=0x7f8b6410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/2519428336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b6410c590 0x7f8b6410e9c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009ca0 tx=0x7f8b5c02a950 comp rx=0 tx=0).stop 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/2519428336 shutdown_connections 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/2519428336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b6410c590 0x7f8b6410e9c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/2519428336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/2519428336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b641031c0 0x7f8b641035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.460+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/2519428336 >> 172.21.15.32:0/2519428336 conn(0x7f8b64077f70 msgr2=0x7f8b64078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.461+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/2519428336 shutdown_connections 2024-04-23T18:16:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.461+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/2519428336 wait complete. 2024-04-23T18:16:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.461+0000 7f8b6c3e8640 1 Processor -- start 2024-04-23T18:16:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.461+0000 7f8b6c3e8640 1 -- start start 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6410c590 0x7f8b64108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64102310 con 0x7f8b641031c0 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64102190 con 0x7f8b64103ae0 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6c3e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64102490 con 0x7f8b6410c590 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59104/0 (socket says 172.21.15.32:59104) 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 -- 172.21.15.32:0/3367724313 learned_addr learned my addr 172.21.15.32:0/3367724313 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b69983640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a985640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6410c590 0x7f8b64108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 -- 172.21.15.32:0/3367724313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6410c590 msgr2=0x7f8b64108cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6410c590 0x7f8b64108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 -- 172.21.15.32:0/3367724313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 msgr2=0x7f8b6410bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 -- 172.21.15.32:0/3367724313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641095b0 con 0x7f8b641031c0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b69983640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.462+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e9f0 tx=0x7f8b5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.463+0000 7f8b5b7fe640 1 -- 172.21.15.32:0/3367724313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400cd80 con 0x7f8b641031c0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.463+0000 7f8b5b7fe640 1 -- 172.21.15.32:0/3367724313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b540108e0 con 0x7f8b641031c0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.463+0000 7f8b5b7fe640 1 -- 172.21.15.32:0/3367724313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400cd80 con 0x7f8b641031c0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.463+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b641098a0 con 0x7f8b641031c0 2024-04-23T18:16:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.463+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641ab7a0 con 0x7f8b641031c0 2024-04-23T18:16:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.467+0000 7f8b5b7fe640 1 -- 172.21.15.32:0/3367724313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8b54010430 con 0x7f8b641031c0 2024-04-23T18:16:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.468+0000 7f8b5b7fe640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 0x7f8b500a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.468+0000 7f8b6f1ce740 1 Processor -- start 2024-04-23T18:16:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.468+0000 7f8b5b7fe640 1 -- 172.21.15.32:0/3367724313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8b540d0770 con 0x7f8b641031c0 2024-04-23T18:16:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.468+0000 7f8b69983640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 0x7f8b500a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 -- start start 2024-04-23T18:16:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9eba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b69983640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 0x7f8b500a6410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b6410a2b0 tx=0x7f8b60005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55571f950970 con 0x55571f9f6800 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55571f8ed4a0 con 0x55571f9ebf40 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.469+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55571f94c930 con 0x55571f9eb620 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59120/0 (socket says 172.21.15.32:59120) 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 -- 172.21.15.32:0/1013835521 learned_addr learned my addr 172.21.15.32:0/1013835521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b69983640 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a184640 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 -- 172.21.15.32:0/1013835521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 msgr2=0x55571f9eba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9eba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 -- 172.21.15.32:0/1013835521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 msgr2=0x55571f9f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 -- 172.21.15.32:0/1013835521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55571f663210 con 0x55571f9f6800 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.470+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c02aec0 tx=0x7f8b5c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8d2598c70e902 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.471+0000 7f8b58bfe640 1 -- 172.21.15.32:0/1013835521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c034050 con 0x55571f9f6800 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.471+0000 7f8b58bfe640 1 -- 172.21.15.32:0/1013835521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5c003e20 con 0x55571f9f6800 2024-04-23T18:16:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.471+0000 7f8b58bfe640 1 -- 172.21.15.32:0/1013835521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0042a0 con 0x55571f9f6800 2024-04-23T18:16:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1013835521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 msgr2=0x55571f9f8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c02aec0 tx=0x7f8b5c009a20 comp rx=0 tx=0).stop 2024-04-23T18:16:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1013835521 shutdown_connections 2024-04-23T18:16:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f8bf0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1013835521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9eba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1013835521 >> 172.21.15.32:0/1013835521 conn(0x55571f9e7b30 msgr2=0x55571f9e7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1013835521 shutdown_connections 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.472+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1013835521 wait complete. 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 -- start start 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 Processor -- start 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9f6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b69983640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b69983640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38026/0 (socket says 172.21.15.32:38026) 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b69983640 1 -- 172.21.15.32:0/3693559704 learned_addr learned my addr 172.21.15.32:0/3693559704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55571f9f93d0 con 0x55571f9f6800 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55571f8eb0b0 con 0x55571f9ebf40 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.473+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55571f9f9550 con 0x55571f9eb620 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b6a985640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b69983640 1 -- 172.21.15.32:0/3693559704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 msgr2=0x55571f9f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b69983640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b69983640 1 -- 172.21.15.32:0/3693559704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 msgr2=0x55571f9f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b69983640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b69983640 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571f72f620 con 0x55571f9ebf40 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b6f1ce740 10 client.0 _ll_register_callbacks cb 0x55571f8d8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.474+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b5c003e20 con 0x55571f9ebf40 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.475+0000 7f8b69983640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b60003970 tx=0x7f8b60004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3f4ddd2527f9da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.475+0000 7f8b4dffb640 10 client.25436 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:16:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.475+0000 7f8b4dffb640 1 -- 172.21.15.32:0/3693559704 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000e7e0 con 0x55571f9ebf40 2024-04-23T18:16:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.475+0000 7f8b4dffb640 1 -- 172.21.15.32:0/3693559704 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8b6000e980 con 0x55571f9ebf40 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.476+0000 7f8b4dffb640 1 -- 172.21.15.32:0/3693559704 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b6000ec00 con 0x55571f9ebf40 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.476+0000 7f8b6f1ce740 10 client.25436 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.476+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55571fa051d0 con 0x55571f9ebf40 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.476+0000 7f8b4dffb640 1 -- 172.21.15.32:0/3693559704 <== mon.2 v2:172.21.15.40:3300/0 4 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f8b60031470 con 0x55571f9ebf40 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 10 client.25436 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 10 client.25436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 20 client.25436 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa08af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 4 client.25436 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55571f95e390 con 0x55571fa06660 2024-04-23T18:16:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa08af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b6a184640 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa08af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ca54ef9ff3479f4 server_cookie=84820d4eda6957a7 in_seq=0 out_seq=0 2024-04-23T18:16:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.477+0000 7f8b4dffb640 10 client.25436 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.478+0000 7f8b4dffb640 1 -- 172.21.15.32:0/3693559704 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55571f95e390 con 0x55571fa06660 2024-04-23T18:16:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.478+0000 7f8b4dffb640 10 client.25436 handle_command_reply: tid=0 2024-04-23T18:16:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.478+0000 7f8b6f1ce740 1 client.25436 shutdown 2024-04-23T18:16:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 msgr2=0x55571f9f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b60003970 tx=0x7f8b60004730 comp rx=0 tx=0).stop 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 shutdown_connections 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa08af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9f6800 0x55571f9f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b4dffb640 0 client.25436 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9ebf40 0x55571f9f3440 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.480+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/3693559704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9eb620 0x55571f9f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.481+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 >> 172.21.15.32:0/3693559704 conn(0x55571f9e7b30 msgr2=0x55571f9ed860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.481+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 shutdown_connections 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.481+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/3693559704 wait complete. 2024-04-23T18:16:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.481+0000 7f8b6f1ce740 20 client.25436 trim_cache size 0 max 16384 2024-04-23T18:16:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 Processor -- start 2024-04-23T18:16:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 -- start start 2024-04-23T18:16:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9eb620 0x55571fa02290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9ebf40 0x55571f9e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b6a60 con 0x55571f9eb620 2024-04-23T18:16:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b42d0 con 0x55571f9f6800 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.503+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b2120 con 0x55571f9ebf40 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.504+0000 7f8b6a985640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.504+0000 7f8b6a985640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38042/0 (socket says 172.21.15.32:38042) 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 -- 172.21.15.32:0/1404195813 learned_addr learned my addr 172.21.15.32:0/1404195813 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 -- 172.21.15.32:0/1404195813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9ebf40 msgr2=0x55571f9e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9ebf40 0x55571f9e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 -- 172.21.15.32:0/1404195813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9eb620 msgr2=0x55571fa02290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9eb620 0x55571fa02290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.505+0000 7f8b6a985640 1 -- 172.21.15.32:0/1404195813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55571f9f23f0 con 0x55571f9f6800 2024-04-23T18:16:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.506+0000 7f8b6a985640 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c02af40 tx=0x7f8b5c02b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe695dd5759658e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.507+0000 7f8b4ffff640 1 -- 172.21.15.32:0/1404195813 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c04f070 con 0x55571f9f6800 2024-04-23T18:16:56.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.507+0000 7f8b4ffff640 1 -- 172.21.15.32:0/1404195813 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5c02f420 con 0x55571f9f6800 2024-04-23T18:16:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.508+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1404195813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 msgr2=0x55571fa0e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.508+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c02af40 tx=0x7f8b5c02b050 comp rx=0 tx=0).stop 2024-04-23T18:16:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.511+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1404195813 shutdown_connections 2024-04-23T18:16:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.511+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9f6800 0x55571fa0e400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.511+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9ebf40 0x55571f9e6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.511+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/1404195813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9eb620 0x55571fa02290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.511+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1404195813 >> 172.21.15.32:0/1404195813 conn(0x55571f9e7b30 msgr2=0x55571f9e7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.512+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1404195813 shutdown_connections 2024-04-23T18:16:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.513+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/1404195813 wait complete. 2024-04-23T18:16:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 -- start start 2024-04-23T18:16:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 Processor -- start 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9eb620 0x55571f9fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 0x55571f9fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 0x55571f9fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b2790 con 0x55571f9ebf40 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b3c00 con 0x55571f9eb620 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.514+0000 7f8b6f1ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641b2e60 con 0x55571f9f6800 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.515+0000 7f8b6f1ce740 10 client.0 _ll_register_callbacks cb 0x7f8b600035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.515+0000 7f8b69983640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 0x55571f9fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.515+0000 7f8b69983640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 0x55571f9fc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59150/0 (socket says 172.21.15.32:59150) 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.515+0000 7f8b69983640 1 -- 172.21.15.32:0/275537436 learned_addr learned my addr 172.21.15.32:0/275537436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:16:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.515+0000 7f8b6a985640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 0x55571f9fcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 -- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9eb620 msgr2=0x55571f9fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9eb620 0x55571f9fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 -- 172.21.15.32:0/275537436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 msgr2=0x55571f9fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 0x55571f9fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 -- 172.21.15.32:0/275537436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571f9ebd70 con 0x55571f9f6800 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b6a985640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 0x55571f9fcd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009bf0 tx=0x7f8b5c02f5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=638645d87a8fa86a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.516+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c04f040 con 0x55571f9f6800 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8b5c02f870 con 0x55571f9f6800 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b5c02f420 con 0x55571f9f6800 2024-04-23T18:16:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b5c02fa70 con 0x55571f9f6800 2024-04-23T18:16:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b6f1ce740 10 client.25423 fetch_fsmap learned FSMap version 26 2024-04-23T18:16:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55571fa7de60 con 0x55571f9f6800 2024-04-23T18:16:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.517+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5c02f420 con 0x55571f9f6800 2024-04-23T18:16:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f8b5c041070 con 0x55571f9f6800 2024-04-23T18:16:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 10 client.25423 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 10 client.25423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 20 client.25423 populate_metadata read hostname 'smithi032' 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 4 client.25423 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55571f958200 con 0x55571fa06660 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.518+0000 7f8b6a184640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.519+0000 7f8b4dffb640 10 client.25423 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.519+0000 7f8b6a184640 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa800f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61c8fc509d37cd51 server_cookie=1cee115dafb47e80 in_seq=0 out_seq=0 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b4dffb640 1 -- 172.21.15.32:0/275537436 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55571f958200 con 0x55571fa06660 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b4dffb640 10 client.25423 handle_command_reply: tid=0 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 client.25423 shutdown 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 msgr2=0x55571f9fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 0x55571f9fcd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009bf0 tx=0x7f8b5c02f5d0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 shutdown_connections 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55571fa06660 0x55571fa800f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55571f9f6800 0x55571f9fcd10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55571f9ebf40 0x55571f9fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 --2- 172.21.15.32:0/275537436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55571f9eb620 0x55571f9fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 >> 172.21.15.32:0/275537436 conn(0x55571f9e7b30 msgr2=0x55571f9e6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.529+0000 7f8b4dffb640 0 client.25423 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.530+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 shutdown_connections 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.530+0000 7f8b6f1ce740 1 -- 172.21.15.32:0/275537436 wait complete. 2024-04-23T18:16:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.530+0000 7f8b6f1ce740 20 client.25423 trim_cache size 0 max 16384 2024-04-23T18:16:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:16:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:16:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:16:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 msgr2=0x7f8b500a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 0x7f8b500a6410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b6410a2b0 tx=0x7f8b60005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 msgr2=0x7f8b6410ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e9f0 tx=0x7f8b5400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 shutdown_connections 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6410c590 0x7f8b64108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b64103ae0 0x7f8b6410bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8b500a3f50 0x7f8b500a6410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 --2- 172.21.15.32:0/3367724313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b641031c0 0x7f8b6410ba50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.531+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 >> 172.21.15.32:0/3367724313 conn(0x7f8b64077f70 msgr2=0x7f8b640fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.532+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 shutdown_connections 2024-04-23T18:16:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:16:56.532+0000 7f8b6c3e8640 1 -- 172.21.15.32:0/3367724313 wait complete. 2024-04-23T18:16:57.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:56 smithi032 ceph-mon[29221]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 674 op/s 2024-04-23T18:16:57.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:56 smithi040 ceph-mon[32626]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 674 op/s 2024-04-23T18:16:57.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:56 smithi064 ceph-mon[33062]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 674 op/s 2024-04-23T18:16:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 90.0 s, 3572.2 tps, lat 27.902 ms stddev 29.906 2024-04-23T18:16:59.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:16:58 smithi032 ceph-mon[29221]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 643 op/s 2024-04-23T18:16:59.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:16:58 smithi040 ceph-mon[32626]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 643 op/s 2024-04-23T18:16:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:16:58 smithi064 ceph-mon[33062]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 643 op/s 2024-04-23T18:17:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:00 smithi032 ceph-mon[29221]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 MiB/s wr, 961 op/s 2024-04-23T18:17:01.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:00 smithi040 ceph-mon[32626]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 MiB/s wr, 961 op/s 2024-04-23T18:17:01.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:00 smithi064 ceph-mon[33062]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 MiB/s wr, 961 op/s 2024-04-23T18:17:03.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:02 smithi032 ceph-mon[29221]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 651 op/s 2024-04-23T18:17:03.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:02 smithi040 ceph-mon[32626]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 651 op/s 2024-04-23T18:17:03.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:02 smithi064 ceph-mon[33062]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 651 op/s 2024-04-23T18:17:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 95.0 s, 1951.2 tps, lat 50.879 ms stddev 70.919 2024-04-23T18:17:05.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:04 smithi032 ceph-mon[29221]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 987 op/s 2024-04-23T18:17:05.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:04 smithi040 ceph-mon[32626]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 987 op/s 2024-04-23T18:17:05.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:04 smithi064 ceph-mon[33062]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 987 op/s 2024-04-23T18:17:07.223 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:06 smithi064 ceph-mon[33062]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 727 op/s 2024-04-23T18:17:07.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:06 smithi032 ceph-mon[29221]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 727 op/s 2024-04-23T18:17:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:06 smithi040 ceph-mon[32626]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 727 op/s 2024-04-23T18:17:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 100.0 s, 1892.2 tps, lat 53.119 ms stddev 94.170 2024-04-23T18:17:09.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:08 smithi032 ceph-mon[29221]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 MiB/s wr, 697 op/s 2024-04-23T18:17:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:08 smithi040 ceph-mon[32626]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 MiB/s wr, 697 op/s 2024-04-23T18:17:09.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:08 smithi064 ceph-mon[33062]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 MiB/s wr, 697 op/s 2024-04-23T18:17:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 -- 172.21.15.32:0/2922755887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408108d90 msgr2=0x7fa408109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 --2- 172.21.15.32:0/2922755887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408108d90 0x7fa408109210 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009930 tx=0x7fa3f802a420 comp rx=0 tx=0).stop 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 -- 172.21.15.32:0/2922755887 shutdown_connections 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 --2- 172.21.15.32:0/2922755887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa408109750 0x7fa408115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 --2- 172.21.15.32:0/2922755887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408108d90 0x7fa408109210 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 --2- 172.21.15.32:0/2922755887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa40810e6c0 0x7fa40810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 -- 172.21.15.32:0/2922755887 >> 172.21.15.32:0/2922755887 conn(0x7fa408077ec0 msgr2=0x7fa40807a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.660+0000 7fa41079b640 1 -- 172.21.15.32:0/2922755887 shutdown_connections 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.661+0000 7fa41079b640 1 -- 172.21.15.32:0/2922755887 wait complete. 2024-04-23T18:17:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.661+0000 7fa41079b640 1 Processor -- start 2024-04-23T18:17:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.662+0000 7fa41079b640 1 -- start start 2024-04-23T18:17:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408109750 0x7fa40807d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa40810e6c0 0x7fa40807dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081187f0 con 0x7fa408109750 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa408118670 con 0x7fa408108d90 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa41079b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa408118970 con 0x7fa40810e6c0 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36688/0 (socket says 172.21.15.32:36688) 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 -- 172.21.15.32:0/2968695537 learned_addr learned my addr 172.21.15.32:0/2968695537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40dd36640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408109750 0x7fa40807d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40ed38640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa40810e6c0 0x7fa40807dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 -- 172.21.15.32:0/2968695537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa40810e6c0 msgr2=0x7fa40807dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa40810e6c0 0x7fa40807dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 -- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408109750 msgr2=0x7fa40807d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408109750 0x7fa40807d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.663+0000 7fa40e537640 1 -- 172.21.15.32:0/2968695537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40807e290 con 0x7fa408108d90 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa40e537640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa40000e9f0 tx=0x7fa40000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/2968695537 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40000cd80 con 0x7fa408108d90 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/2968695537 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4000108e0 con 0x7fa408108d90 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa408082110 con 0x7fa408108d90 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/2968695537 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400018980 con 0x7fa408108d90 2024-04-23T18:17:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.664+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa408082630 con 0x7fa408108d90 2024-04-23T18:17:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.667+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/2968695537 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa400018b20 con 0x7fa408108d90 2024-04-23T18:17:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.667+0000 7fa413581740 1 Processor -- start 2024-04-23T18:17:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 -- start start 2024-04-23T18:17:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8620 0x55d66f7e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7f3800 0x55d66f7f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d66f74d970 con 0x55d66f7e8620 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d66f6ea4a0 con 0x55d66f7f3800 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa413581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d66f749930 con 0x55d66f7e8f40 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.668+0000 7fa3ff7fe640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 0x7fa3f40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40e537640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40e537640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35018/0 (socket says 172.21.15.32:35018) 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40e537640 1 -- 172.21.15.32:0/583096874 learned_addr learned my addr 172.21.15.32:0/583096874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/2968695537 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4000d0c10 con 0x7fa408108d90 2024-04-23T18:17:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40dd36640 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7f3800 0x55d66f7f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40ed38640 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8620 0x55d66f7e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.669+0000 7fa40ed38640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 0x7fa3f40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.670+0000 7fa40e537640 1 -- 172.21.15.32:0/583096874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7f3800 msgr2=0x55d66f7f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.670+0000 7fa40e537640 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7f3800 0x55d66f7f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.670+0000 7fa40e537640 1 -- 172.21.15.32:0/583096874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8620 msgr2=0x55d66f7e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.670+0000 7fa40e537640 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8620 0x55d66f7e8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.670+0000 7fa40e537640 1 -- 172.21.15.32:0/583096874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d66f460210 con 0x55d66f7e8f40 2024-04-23T18:17:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.671+0000 7fa40e537640 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa40009a4f0 tx=0x7fa400013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c98fbda8d19fba5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.672+0000 7fa40ed38640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 0x7fa3f40a6250 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa404005c40 tx=0x7fa404009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.672+0000 7fa3fcff9640 1 -- 172.21.15.32:0/583096874 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400013190 con 0x55d66f7e8f40 2024-04-23T18:17:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.672+0000 7fa3fcff9640 1 -- 172.21.15.32:0/583096874 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa40009ee50 con 0x55d66f7e8f40 2024-04-23T18:17:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.672+0000 7fa3fcff9640 1 -- 172.21.15.32:0/583096874 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4000ce5d0 con 0x55d66f7e8f40 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.673+0000 7fa413581740 1 -- 172.21.15.32:0/583096874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 msgr2=0x55d66f7f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.673+0000 7fa413581740 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa40009a4f0 tx=0x7fa400013000 comp rx=0 tx=0).stop 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 -- 172.21.15.32:0/583096874 shutdown_connections 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7f3800 0x55d66f7f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f32c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 --2- 172.21.15.32:0/583096874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8620 0x55d66f7e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 -- 172.21.15.32:0/583096874 >> 172.21.15.32:0/583096874 conn(0x55d66f7e4b30 msgr2=0x55d66f7e4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 -- 172.21.15.32:0/583096874 shutdown_connections 2024-04-23T18:17:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.675+0000 7fa413581740 1 -- 172.21.15.32:0/583096874 wait complete. 2024-04-23T18:17:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.676+0000 7fa413581740 1 -- start start 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.676+0000 7fa413581740 1 Processor -- start 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.676+0000 7fa413581740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.676+0000 7fa413581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.676+0000 7fa413581740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa413581740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d66f7f63d0 con 0x55d66f7f3800 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa413581740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d66f6e80b0 con 0x55d66f7e8620 2024-04-23T18:17:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa413581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d66f7f6550 con 0x55d66f7e8f40 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa413581740 10 client.0 _ll_register_callbacks cb 0x55d66f6d5d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33774/0 (socket says 172.21.15.32:33774) 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 -- 172.21.15.32:0/3375114331 learned_addr learned my addr 172.21.15.32:0/3375114331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40e537640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40dd36640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 -- 172.21.15.32:0/3375114331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 msgr2=0x55d66f7f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 -- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 msgr2=0x55d66f7f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.677+0000 7fa40ed38640 1 -- 172.21.15.32:0/3375114331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d66f52c620 con 0x55d66f7f3800 2024-04-23T18:17:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.678+0000 7fa40e537640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.678+0000 7fa40ed38640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa404012720 tx=0x7fa404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a487dcb395385c58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.679+0000 7fa3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.679+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404004230 con 0x55d66f7f3800 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.679+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa40009ee50 con 0x55d66f7f3800 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.679+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4040043d0 con 0x55d66f7f3800 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.680+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa4040045d0 con 0x55d66f7f3800 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.680+0000 7fa413581740 10 client.15807 fetch_fsmap learned FSMap version 26 2024-04-23T18:17:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.680+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d66f8023e0 con 0x55d66f7f3800 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.680+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa404024070 con 0x55d66f7f3800 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fa40401f410 con 0x55d66f7f3800 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 10 client.15807 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 20 client.15807 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f803c70 0x55d66f8060a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 4 client.15807 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.681+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa40009ee50 con 0x55d66f803c70 2024-04-23T18:17:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.686+0000 7fa40e537640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f803c70 0x55d66f8060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.691+0000 7fa40e537640 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f803c70 0x55d66f8060a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460d66db1b831ac2 server_cookie=a9e1c78a24620756 in_seq=0 out_seq=0 2024-04-23T18:17:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.691+0000 7fa3f1ffb640 10 client.15807 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.692+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/3375114331 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa40009ee50 con 0x55d66f803c70 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.692+0000 7fa3f1ffb640 10 client.15807 handle_command_reply: tid=0 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.692+0000 7fa413581740 1 client.15807 shutdown 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.692+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 msgr2=0x55d66f7f0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.692+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa404012720 tx=0x7fa404004a40 comp rx=0 tx=0).stop 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 shutdown_connections 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f803c70 0x55d66f8060a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f7f0af0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 --2- 172.21.15.32:0/3375114331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 >> 172.21.15.32:0/3375114331 conn(0x55d66f7e4b30 msgr2=0x55d66f7ea8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa3f1ffb640 0 client.15807 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 shutdown_connections 2024-04-23T18:17:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.693+0000 7fa413581740 1 -- 172.21.15.32:0/3375114331 wait complete. 2024-04-23T18:17:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.694+0000 7fa413581740 20 client.15807 trim_cache size 0 max 16384 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.720+0000 7fa413581740 1 Processor -- start 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.720+0000 7fa413581740 1 -- start start 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.720+0000 7fa413581740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.720+0000 7fa413581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.720+0000 7fa413581740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f80b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa413581740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081bf3e0 con 0x55d66f7f3800 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa413581740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081bcc50 con 0x55d66f7e8620 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa413581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081baaa0 con 0x55d66f7e8f40 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36702/0 (socket says 172.21.15.32:36702) 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 -- 172.21.15.32:0/4038317807 learned_addr learned my addr 172.21.15.32:0/4038317807 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40dd36640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40ed38640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f80b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 -- 172.21.15.32:0/4038317807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 msgr2=0x55d66f7e3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7e3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 -- 172.21.15.32:0/4038317807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 msgr2=0x55d66f80b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f80b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 -- 172.21.15.32:0/4038317807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d66f7e02e0 con 0x55d66f7e8620 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40dd36640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7e3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.721+0000 7fa40e537640 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c6a0 tx=0x7fa400000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8953d7f599f21f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.722+0000 7fa3f3fff640 1 -- 172.21.15.32:0/4038317807 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4000a6790 con 0x55d66f7e8620 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.722+0000 7fa3f3fff640 1 -- 172.21.15.32:0/4038317807 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa40009ee50 con 0x55d66f7e8620 2024-04-23T18:17:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.722+0000 7fa3f3fff640 1 -- 172.21.15.32:0/4038317807 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4000ce5d0 con 0x55d66f7e8620 2024-04-23T18:17:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 -- 172.21.15.32:0/4038317807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 msgr2=0x55d66f7ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c6a0 tx=0x7fa400000bc0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 -- 172.21.15.32:0/4038317807 shutdown_connections 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7f3800 0x55d66f80b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7e8f40 0x55d66f7e3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 --2- 172.21.15.32:0/4038317807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7ff290 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.723+0000 7fa413581740 1 -- 172.21.15.32:0/4038317807 >> 172.21.15.32:0/4038317807 conn(0x55d66f7e4b30 msgr2=0x55d66f7e3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.724+0000 7fa413581740 1 -- 172.21.15.32:0/4038317807 shutdown_connections 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.724+0000 7fa413581740 1 -- 172.21.15.32:0/4038317807 wait complete. 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 -- start start 2024-04-23T18:17:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 Processor -- start 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8f40 0x55d66f7f9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7f3800 0x55d66f7f9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa40e537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36710/0 (socket says 172.21.15.32:36710) 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa40e537640 1 -- 172.21.15.32:0/1372955942 learned_addr learned my addr 172.21.15.32:0/1372955942 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081bb110 con 0x55d66f7e8f40 2024-04-23T18:17:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081bc580 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081bb7e0 con 0x55d66f7f3800 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.725+0000 7fa40ed38640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7f3800 0x55d66f7f9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa413581740 10 client.0 _ll_register_callbacks cb 0x7fa404003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 -- 172.21.15.32:0/1372955942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7f3800 msgr2=0x55d66f7f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7f3800 0x55d66f7f9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 -- 172.21.15.32:0/1372955942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8f40 msgr2=0x55d66f7f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8f40 0x55d66f7f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d66f869230 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa40e537640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa4000a6260 tx=0x7fa4000a6800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17fdda6ad6263e31 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4000bb060 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.726+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4000bd0b0 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa40009ee50 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa400067050 con 0x55d66f7e8620 2024-04-23T18:17:09.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 10 client.25457 fetch_fsmap learned FSMap version 26 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d66f86a440 con 0x55d66f7e8620 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa40009a190 con 0x55d66f7e8620 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fa400062b60 con 0x55d66f7e8620 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 10 client.25457 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 20 client.25457 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f86a660 0x55d66f86ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 4 client.25457 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.727+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa40009ee50 con 0x55d66f86a660 2024-04-23T18:17:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.728+0000 7fa40dd36640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f86a660 0x55d66f86ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.728+0000 7fa40dd36640 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f86a660 0x55d66f86ca50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56d2c794f5104fba server_cookie=9954ab3726194d5e in_seq=0 out_seq=0 2024-04-23T18:17:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.729+0000 7fa3f1ffb640 10 client.25457 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa3f1ffb640 1 -- 172.21.15.32:0/1372955942 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa40009ee50 con 0x55d66f86a660 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa3f1ffb640 10 client.25457 handle_command_reply: tid=0 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 client.25457 shutdown 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 msgr2=0x55d66f7f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa4000a6260 tx=0x7fa4000a6800 comp rx=0 tx=0).stop 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 shutdown_connections 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d66f86a660 0x55d66f86ca50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa3f1ffb640 0 client.25457 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d66f7f3800 0x55d66f7f9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d66f7e8f40 0x55d66f7f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 --2- 172.21.15.32:0/1372955942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d66f7e8620 0x55d66f7f9130 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa4000a6260 tx=0x7fa4000a6800 comp rx=0 tx=0).stop 2024-04-23T18:17:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.742+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 >> 172.21.15.32:0/1372955942 conn(0x55d66f7e4b30 msgr2=0x55d66f7e3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.743+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 shutdown_connections 2024-04-23T18:17:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.743+0000 7fa413581740 1 -- 172.21.15.32:0/1372955942 wait complete. 2024-04-23T18:17:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.743+0000 7fa413581740 20 client.25457 trim_cache size 0 max 16384 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 msgr2=0x7fa3f40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 0x7fa3f40a6250 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa404005c40 tx=0x7fa404009500 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 msgr2=0x7fa4080845f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa40000e9f0 tx=0x7fa40000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 shutdown_connections 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa40810e6c0 0x7fa40807dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3f40a3d90 0x7fa3f40a6250 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa408109750 0x7fa40807d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 --2- 172.21.15.32:0/2968695537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108d90 0x7fa4080845f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.744+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 >> 172.21.15.32:0/2968695537 conn(0x7fa408077ec0 msgr2=0x7fa40807a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.746+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 shutdown_connections 2024-04-23T18:17:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:09.746+0000 7fa41079b640 1 -- 172.21.15.32:0/2968695537 wait complete. 2024-04-23T18:17:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:10 smithi032 ceph-mon[29221]: pgmap v65: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 2.07k op/s 2024-04-23T18:17:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:10 smithi040 ceph-mon[32626]: pgmap v65: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 2.07k op/s 2024-04-23T18:17:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:10 smithi064 ceph-mon[33062]: pgmap v65: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 2.07k op/s 2024-04-23T18:17:13.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:12 smithi032 ceph-mon[29221]: pgmap v66: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-23T18:17:13.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:12 smithi040 ceph-mon[32626]: pgmap v66: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-23T18:17:13.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:12 smithi064 ceph-mon[33062]: pgmap v66: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-23T18:17:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 105.0 s, 3668.4 tps, lat 27.340 ms stddev 28.878 2024-04-23T18:17:14.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:13 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:13 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:13 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:14 smithi032 ceph-mon[29221]: pgmap v67: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 2.21k op/s 2024-04-23T18:17:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:14 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:14 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:14 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:14 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:14 smithi040 ceph-mon[32626]: pgmap v67: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 2.21k op/s 2024-04-23T18:17:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:14 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:14 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:14 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:14 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:14 smithi064 ceph-mon[33062]: pgmap v67: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 2.21k op/s 2024-04-23T18:17:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:14 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:14 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:14 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:14 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:16.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:15 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:16.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:15 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:16.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:15 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:16.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:15 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:15 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:16.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:15 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:16.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:15 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:16.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:15 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:16.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:15 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:17:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:16 smithi032 ceph-mon[29221]: pgmap v68: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 1.95k op/s 2024-04-23T18:17:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:16 smithi040 ceph-mon[32626]: pgmap v68: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 1.95k op/s 2024-04-23T18:17:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:16 smithi064 ceph-mon[33062]: pgmap v68: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 1.95k op/s 2024-04-23T18:17:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 110.0 s, 3899.0 tps, lat 25.656 ms stddev 27.602 2024-04-23T18:17:19.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:18 smithi032 ceph-mon[29221]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 37 MiB/s wr, 1.90k op/s 2024-04-23T18:17:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:18 smithi040 ceph-mon[32626]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 37 MiB/s wr, 1.90k op/s 2024-04-23T18:17:19.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:18 smithi064 ceph-mon[33062]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 37 MiB/s wr, 1.90k op/s 2024-04-23T18:17:20.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:19 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:20.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:19 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:20.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:19 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:20 smithi032 ceph-mon[29221]: pgmap v70: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 41 MiB/s wr, 2.22k op/s 2024-04-23T18:17:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:20 smithi040 ceph-mon[32626]: pgmap v70: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 41 MiB/s wr, 2.22k op/s 2024-04-23T18:17:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:20 smithi064 ceph-mon[33062]: pgmap v70: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 41 MiB/s wr, 2.22k op/s 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 -- 172.21.15.32:0/1693961106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1660107280 msgr2=0x7f166010ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 --2- 172.21.15.32:0/1693961106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1660107280 0x7f166010ee50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1658009900 tx=0x7f165802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 -- 172.21.15.32:0/1693961106 shutdown_connections 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 --2- 172.21.15.32:0/1693961106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1660107280 0x7f166010ee50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 --2- 172.21.15.32:0/1693961106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660102b60 0x7f1660106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 --2- 172.21.15.32:0/1693961106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 -- 172.21.15.32:0/1693961106 >> 172.21.15.32:0/1693961106 conn(0x7f16600fda30 msgr2=0x7f16600ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 -- 172.21.15.32:0/1693961106 shutdown_connections 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.866+0000 7f166522a640 1 -- 172.21.15.32:0/1693961106 wait complete. 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 Processor -- start 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 -- start start 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660107280 0x7f1660198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16601121d0 con 0x7f166019d010 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1660112050 con 0x7f16601021b0 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f166522a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1660112350 con 0x7f1660107280 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49014/0 (socket says 172.21.15.32:49014) 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 -- 172.21.15.32:0/4290483555 learned_addr learned my addr 172.21.15.32:0/4290483555 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165e59c640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660107280 0x7f1660198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165ed9d640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 -- 172.21.15.32:0/4290483555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660107280 msgr2=0x7f1660198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660107280 0x7f1660198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 -- 172.21.15.32:0/4290483555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 msgr2=0x7f1660198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.867+0000 7f165f59e640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.868+0000 7f165f59e640 1 -- 172.21.15.32:0/4290483555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166019db00 con 0x7f166019d010 2024-04-23T18:17:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.868+0000 7f165ed9d640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.868+0000 7f165f59e640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1658004770 tx=0x7f1658004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.869+0000 7f1657fff640 1 -- 172.21.15.32:0/4290483555 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165803f030 con 0x7f166019d010 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.869+0000 7f1657fff640 1 -- 172.21.15.32:0/4290483555 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1658003aa0 con 0x7f166019d010 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.869+0000 7f1657fff640 1 -- 172.21.15.32:0/4290483555 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658035070 con 0x7f166019d010 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.869+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166019dd90 con 0x7f166019d010 2024-04-23T18:17:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.869+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16600743c0 con 0x7f166019d010 2024-04-23T18:17:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.872+0000 7f1657fff640 1 -- 172.21.15.32:0/4290483555 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1658042030 con 0x7f166019d010 2024-04-23T18:17:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 Processor -- start 2024-04-23T18:17:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 -- start start 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebcabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561b2eb22970 con 0x561b2ebbd620 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b2eabf4a0 con 0x561b2ebc8800 2024-04-23T18:17:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f166801c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561b2eb1e930 con 0x561b2ebbdf40 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.873+0000 7f1657fff640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 0x7f164c0aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49022/0 (socket says 172.21.15.32:49022) 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 -- 172.21.15.32:0/823405786 learned_addr learned my addr 172.21.15.32:0/823405786 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165f59e640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 0x7f164c0aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebcabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f1657fff640 1 -- 172.21.15.32:0/4290483555 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f16580f0c40 con 0x7f166019d010 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 -- 172.21.15.32:0/823405786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 msgr2=0x561b2ebc82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 -- 172.21.15.32:0/823405786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 msgr2=0x561b2ebcabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebcabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.874+0000 7f165e59c640 1 -- 172.21.15.32:0/823405786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2e835210 con 0x561b2ebbd620 2024-04-23T18:17:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.875+0000 7f165e59c640 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x561b2ebbacc0 tx=0x7f164800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5217383e9c7f985e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.875+0000 7f165f59e640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 0x7f164c0aa730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16580b9090 tx=0x7f16580b87a0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.875+0000 7f16557fa640 1 -- 172.21.15.32:0/823405786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16480120d0 con 0x561b2ebbd620 2024-04-23T18:17:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.876+0000 7f16557fa640 1 -- 172.21.15.32:0/823405786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16480047a0 con 0x561b2ebbd620 2024-04-23T18:17:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.876+0000 7f16557fa640 1 -- 172.21.15.32:0/823405786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648005290 con 0x561b2ebbd620 2024-04-23T18:17:22.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- 172.21.15.32:0/823405786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 msgr2=0x561b2ebbda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x561b2ebbacc0 tx=0x7f164800b650 comp rx=0 tx=0).stop 2024-04-23T18:17:22.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- 172.21.15.32:0/823405786 shutdown_connections 2024-04-23T18:17:22.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebcabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 --2- 172.21.15.32:0/823405786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebbda00 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- 172.21.15.32:0/823405786 >> 172.21.15.32:0/823405786 conn(0x561b2ebb9b30 msgr2=0x561b2ebb9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- 172.21.15.32:0/823405786 shutdown_connections 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- 172.21.15.32:0/823405786 wait complete. 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 -- start start 2024-04-23T18:17:22.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.877+0000 7f166801c740 1 Processor -- start 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebc5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561b2ebcb3d0 con 0x561b2ebbd620 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b2eabd0b0 con 0x561b2ebc8800 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561b2ebcb550 con 0x561b2ebbdf40 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f166801c740 10 client.0 _ll_register_callbacks cb 0x561b2eaaad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49032/0 (socket says 172.21.15.32:49032) 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 -- 172.21.15.32:0/1442532108 learned_addr learned my addr 172.21.15.32:0/1442532108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebc5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165ed9d640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 -- 172.21.15.32:0/1442532108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 msgr2=0x561b2ebc5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 -- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 msgr2=0x561b2ebc5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebc5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 -- 172.21.15.32:0/1442532108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2e901620 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f165e59c640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1648015040 tx=0x7f164800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b613ab28b49483 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.878+0000 7f16467fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16480043c0 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16480047a0 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1648005440 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1648005640 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 10 client.15822 fetch_fsmap learned FSMap version 27 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b2ebd7350 con 0x561b2ebbd620 2024-04-23T18:17:22.877 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16480047a0 con 0x561b2ebbd620 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f1648010460 con 0x561b2ebbd620 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 10 client.15822 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 20 client.15822 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ebdab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 4 client.15822 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.879+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b2eb30390 con 0x561b2ebd8760 2024-04-23T18:17:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.880+0000 7f165ed9d640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ebdab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.880+0000 7f16467fc640 10 client.15822 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.880+0000 7f165ed9d640 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ebdab50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96d821d833f7002 server_cookie=d6d94f83ce84f507 in_seq=0 out_seq=0 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f16467fc640 1 -- 172.21.15.32:0/1442532108 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b2eb30390 con 0x561b2ebd8760 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f16467fc640 10 client.15822 handle_command_reply: tid=0 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 client.15822 shutdown 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 msgr2=0x561b2ebc8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1648015040 tx=0x7f164800b410 comp rx=0 tx=0).stop 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 shutdown_connections 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ebdab50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebc8800 0x561b2ebc5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f16467fc640 0 client.15822 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebbdf40 0x561b2ebc5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 --2- 172.21.15.32:0/1442532108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebc8140 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 >> 172.21.15.32:0/1442532108 conn(0x561b2ebb9b30 msgr2=0x561b2ebbf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 shutdown_connections 2024-04-23T18:17:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 1 -- 172.21.15.32:0/1442532108 wait complete. 2024-04-23T18:17:22.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.881+0000 7f166801c740 20 client.15822 trim_cache size 0 max 16384 2024-04-23T18:17:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.912+0000 7f166801c740 1 Processor -- start 2024-04-23T18:17:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.913+0000 7f166801c740 1 -- start start 2024-04-23T18:17:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.913+0000 7f166801c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.913+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f165e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49036/0 (socket says 172.21.15.32:49036) 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f165e59c640 1 -- 172.21.15.32:0/3716885804 learned_addr learned my addr 172.21.15.32:0/3716885804 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbdf40 0x561b2ebb8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebe0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16601ae1a0 con 0x561b2ebbd620 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16601ab9e0 con 0x561b2ebbdf40 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16601a9830 con 0x561b2ebc8800 2024-04-23T18:17:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.914+0000 7f165ed9d640 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbdf40 0x561b2ebb8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebe0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 -- 172.21.15.32:0/3716885804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 msgr2=0x561b2ebe0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebe0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 -- 172.21.15.32:0/3716885804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbdf40 msgr2=0x561b2ebb8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbdf40 0x561b2ebb8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.915+0000 7f165e59c640 1 -- 172.21.15.32:0/3716885804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2ebc5c60 con 0x561b2ebbd620 2024-04-23T18:17:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.917+0000 7f165e59c640 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f164800b5a0 tx=0x7f1648012480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a8c454d342b717 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.917+0000 7f1654ff9640 1 -- 172.21.15.32:0/3716885804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648025070 con 0x561b2ebbd620 2024-04-23T18:17:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.917+0000 7f1654ff9640 1 -- 172.21.15.32:0/3716885804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1648016040 con 0x561b2ebbd620 2024-04-23T18:17:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.917+0000 7f1654ff9640 1 -- 172.21.15.32:0/3716885804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648034040 con 0x561b2ebbd620 2024-04-23T18:17:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.918+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 msgr2=0x561b2ebd42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.918+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f164800b5a0 tx=0x7f1648012480 comp rx=0 tx=0).stop 2024-04-23T18:17:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.919+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 shutdown_connections 2024-04-23T18:17:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.919+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebe0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.919+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbdf40 0x561b2ebb8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.919+0000 7f166801c740 1 --2- 172.21.15.32:0/3716885804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbd620 0x561b2ebd42c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.919+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 >> 172.21.15.32:0/3716885804 conn(0x561b2ebb9b30 msgr2=0x561b2ebc5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.924+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 shutdown_connections 2024-04-23T18:17:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.925+0000 7f166801c740 1 -- 172.21.15.32:0/3716885804 wait complete. 2024-04-23T18:17:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.925+0000 7f166801c740 1 -- start start 2024-04-23T18:17:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.925+0000 7f166801c740 1 Processor -- start 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbdf40 0x561b2ebcddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16601a9ea0 con 0x561b2ebbdf40 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16601ab310 con 0x561b2ebbd620 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16601aa570 con 0x561b2ebc8800 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f166801c740 10 client.0 _ll_register_callbacks cb 0x7f1648004280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49042/0 (socket says 172.21.15.32:49042) 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 -- 172.21.15.32:0/2538125058 learned_addr learned my addr 172.21.15.32:0/2538125058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165ed9d640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbdf40 0x561b2ebcddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 -- 172.21.15.32:0/2538125058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 msgr2=0x561b2ebce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 -- 172.21.15.32:0/2538125058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbdf40 msgr2=0x561b2ebcddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.926+0000 7f165e59c640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbdf40 0x561b2ebcddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.927+0000 7f165e59c640 1 -- 172.21.15.32:0/2538125058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2ebce840 con 0x561b2ebbd620 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.928+0000 7f165e59c640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1648006010 tx=0x7f16480122b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9130549333420bf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.928+0000 7f16467fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:17:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.929+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648042060 con 0x561b2ebbd620 2024-04-23T18:17:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.929+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1648010560 con 0x561b2ebbd620 2024-04-23T18:17:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.929+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1648016040 con 0x561b2ebbd620 2024-04-23T18:17:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.929+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1648025050 con 0x561b2ebbd620 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 10 client.25475 fetch_fsmap learned FSMap version 27 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b2ec3c780 con 0x561b2ebbd620 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1648016040 con 0x561b2ebbd620 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f1648030aa0 con 0x561b2ebbd620 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 10 client.25475 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 10 client.25475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 20 client.25475 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ec522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 4 client.25475 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.930+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b2eb2a200 con 0x561b2ebd8760 2024-04-23T18:17:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.931+0000 7f165ed9d640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ec522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.931+0000 7f16467fc640 10 client.25475 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.931+0000 7f165ed9d640 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ec522d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73991598c05196f server_cookie=fceabf913be6aa20 in_seq=0 out_seq=0 2024-04-23T18:17:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f16467fc640 1 -- 172.21.15.32:0/2538125058 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b2eb2a200 con 0x561b2ebd8760 2024-04-23T18:17:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f16467fc640 10 client.25475 handle_command_reply: tid=0 2024-04-23T18:17:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 client.25475 shutdown 2024-04-23T18:17:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 msgr2=0x561b2ebd0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1648006010 tx=0x7f16480122b0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 shutdown_connections 2024-04-23T18:17:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561b2ebd8760 0x561b2ec522d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b2ebc8800 0x561b2ebce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561b2ebbdf40 0x561b2ebcddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 --2- 172.21.15.32:0/2538125058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2ebbd620 0x561b2ebd0840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 >> 172.21.15.32:0/2538125058 conn(0x561b2ebb9b30 msgr2=0x561b2ebd32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f16467fc640 0 client.25475 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 shutdown_connections 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 1 -- 172.21.15.32:0/2538125058 wait complete. 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.943+0000 7f166801c740 20 client.25475 trim_cache size 0 max 16384 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:17:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 msgr2=0x7f164c0aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 0x7f164c0aa730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16580b9090 tx=0x7f16580b87a0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 msgr2=0x7f166019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1658004770 tx=0x7f1658004270 comp rx=0 tx=0).stop 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 shutdown_connections 2024-04-23T18:17:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f164c0a8270 0x7f164c0aa730 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166019d010 0x7f166019d4c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1660107280 0x7f1660198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 --2- 172.21.15.32:0/4290483555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16601021b0 0x7f1660198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 >> 172.21.15.32:0/4290483555 conn(0x7f16600fda30 msgr2=0x7f16600fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 shutdown_connections 2024-04-23T18:17:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:22.944+0000 7f166522a640 1 -- 172.21.15.32:0/4290483555 wait complete. 2024-04-23T18:17:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:22 smithi032 ceph-mon[29221]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.7 MiB/s wr, 850 op/s 2024-04-23T18:17:23.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:22 smithi040 ceph-mon[32626]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.7 MiB/s wr, 850 op/s 2024-04-23T18:17:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:22 smithi064 ceph-mon[33062]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.7 MiB/s wr, 850 op/s 2024-04-23T18:17:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 115.0 s, 3788.2 tps, lat 26.443 ms stddev 26.455 2024-04-23T18:17:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:24 smithi032 ceph-mon[29221]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:17:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:24 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:24 smithi040 ceph-mon[32626]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:17:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:24 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:25.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:24 smithi064 ceph-mon[33062]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:17:25.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:24 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:27.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:26 smithi032 ceph-mon[29221]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-23T18:17:27.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:26 smithi040 ceph-mon[32626]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-23T18:17:27.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:26 smithi064 ceph-mon[33062]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-23T18:17:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 120.0 s, 3994.0 tps, lat 24.935 ms stddev 23.916 2024-04-23T18:17:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:28 smithi032 ceph-mon[29221]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-23T18:17:29.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:28 smithi040 ceph-mon[32626]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-23T18:17:29.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:28 smithi064 ceph-mon[33062]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-23T18:17:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:30 smithi032 ceph-mon[29221]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 977 op/s 2024-04-23T18:17:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:30 smithi040 ceph-mon[32626]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 977 op/s 2024-04-23T18:17:31.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:30 smithi064 ceph-mon[33062]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 977 op/s 2024-04-23T18:17:33.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:32 smithi032 ceph-mon[29221]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 657 op/s 2024-04-23T18:17:33.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:32 smithi040 ceph-mon[32626]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 657 op/s 2024-04-23T18:17:33.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:32 smithi064 ceph-mon[33062]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 657 op/s 2024-04-23T18:17:33.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 125.0 s, 3780.6 tps, lat 26.540 ms stddev 26.815 2024-04-23T18:17:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:35 smithi032 ceph-mon[29221]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 948 op/s 2024-04-23T18:17:35.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:35 smithi040 ceph-mon[32626]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 948 op/s 2024-04-23T18:17:35.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:35 smithi064 ceph-mon[33062]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 948 op/s 2024-04-23T18:17:36.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.087+0000 7fa5c68cf640 1 -- 172.21.15.32:0/214100018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c010e7d0 msgr2=0x7fa5c010ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.087+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/214100018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c010e7d0 0x7fa5c010ec30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009900 tx=0x7fa5b002a440 comp rx=0 tx=0).stop 2024-04-23T18:17:36.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.088+0000 7fa5c68cf640 1 -- 172.21.15.32:0/214100018 shutdown_connections 2024-04-23T18:17:36.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.088+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/214100018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c011b730 0x7fa5c011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.088+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/214100018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c010e7d0 0x7fa5c010ec30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.088+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/214100018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 0x7fa5c010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.088+0000 7fa5c68cf640 1 -- 172.21.15.32:0/214100018 >> 172.21.15.32:0/214100018 conn(0x7fa5c006d030 msgr2=0x7fa5c006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.094+0000 7fa5c68cf640 1 -- 172.21.15.32:0/214100018 shutdown_connections 2024-04-23T18:17:36.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.097+0000 7fa5c68cf640 1 -- 172.21.15.32:0/214100018 wait complete. 2024-04-23T18:17:36.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.098+0000 7fa5c68cf640 1 Processor -- start 2024-04-23T18:17:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.102+0000 7fa5c68cf640 1 -- start start 2024-04-23T18:17:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 0x7fa5c01a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c011b730 0x7fa5c01abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01210a0 con 0x7fa5c011b730 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0120f20 con 0x7fa5c010e7d0 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5c68cf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0121220 con 0x7fa5c01106f0 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:57048/0 (socket says 172.21.15.32:57048) 2024-04-23T18:17:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.103+0000 7fa5bffff640 1 -- 172.21.15.32:0/1179442006 learned_addr learned my addr 172.21.15.32:0/1179442006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 0x7fa5c01a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c011b730 0x7fa5c01abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bffff640 1 -- 172.21.15.32:0/1179442006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 msgr2=0x7fa5c01a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bffff640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 0x7fa5c01a7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bffff640 1 -- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c011b730 msgr2=0x7fa5c01abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bffff640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c011b730 0x7fa5c01abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.104+0000 7fa5bffff640 1 -- 172.21.15.32:0/1179442006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c01ac5d0 con 0x7fa5c010e7d0 2024-04-23T18:17:36.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.106+0000 7fa5bffff640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400b450 tx=0x7fa5b400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.107+0000 7fa5bd7fa640 1 -- 172.21.15.32:0/1179442006 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4015020 con 0x7fa5c010e7d0 2024-04-23T18:17:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.107+0000 7fa5bd7fa640 1 -- 172.21.15.32:0/1179442006 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b4011d40 con 0x7fa5c010e7d0 2024-04-23T18:17:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.107+0000 7fa5bd7fa640 1 -- 172.21.15.32:0/1179442006 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400f340 con 0x7fa5c010e7d0 2024-04-23T18:17:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.108+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c01ac8c0 con 0x7fa5c010e7d0 2024-04-23T18:17:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.108+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c01b4150 con 0x7fa5c010e7d0 2024-04-23T18:17:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.112+0000 7fa5c95eb740 1 Processor -- start 2024-04-23T18:17:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.112+0000 7fa5c95eb740 1 -- start start 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.112+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 0x5562d94aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.112+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 0x5562d94b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.112+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94b9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.113+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5562d9411970 con 0x5562d94acf40 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.113+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562d93ae4a0 con 0x5562d94ac620 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.113+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5562d940d930 con 0x5562d94b7800 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.114+0000 7fa5bd7fa640 1 -- 172.21.15.32:0/1179442006 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa5b4018020 con 0x7fa5c010e7d0 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.114+0000 7fa5c4e6c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 0x5562d94aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.114+0000 7fa5c4e6c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 0x5562d94aca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:57054/0 (socket says 172.21.15.32:57054) 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.114+0000 7fa5c4e6c640 1 -- 172.21.15.32:0/1738036840 learned_addr learned my addr 172.21.15.32:0/1738036840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.115+0000 7fa5bd7fa640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 0x7fa5a80a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.115+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94b9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.115+0000 7fa5bd7fa640 1 -- 172.21.15.32:0/1179442006 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa5b40d0080 con 0x7fa5c010e7d0 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bffff640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 0x5562d94b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/1738036840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 msgr2=0x5562d94aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 0x5562d94aca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/1738036840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 msgr2=0x5562d94b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 0x5562d94b72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/1738036840 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562d9124210 con 0x5562d94b7800 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 0x7fa5a80a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.116+0000 7fa5bffff640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 0x5562d94b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.117+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94b9bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5b002aa30 tx=0x7fa5b0004330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b93e16c651b75de server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.117+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 0x7fa5a80a6360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8008e50 tx=0x7fa5b8004a40 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.117+0000 7fa5a67fc640 1 -- 172.21.15.32:0/1738036840 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0035030 con 0x5562d94b7800 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.117+0000 7fa5a67fc640 1 -- 172.21.15.32:0/1738036840 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b0051070 con 0x5562d94b7800 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.117+0000 7fa5a67fc640 1 -- 172.21.15.32:0/1738036840 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0056a40 con 0x5562d94b7800 2024-04-23T18:17:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 -- 172.21.15.32:0/1738036840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 msgr2=0x5562d94b9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94b9bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5b002aa30 tx=0x7fa5b0004330 comp rx=0 tx=0).stop 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 -- 172.21.15.32:0/1738036840 shutdown_connections 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94b9bf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94acf40 0x5562d94b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/1738036840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94ac620 0x5562d94aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.118+0000 7fa5c95eb740 1 -- 172.21.15.32:0/1738036840 >> 172.21.15.32:0/1738036840 conn(0x5562d94a8b30 msgr2=0x5562d94a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.119+0000 7fa5c95eb740 1 -- 172.21.15.32:0/1738036840 shutdown_connections 2024-04-23T18:17:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.119+0000 7fa5c95eb740 1 -- 172.21.15.32:0/1738036840 wait complete. 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.119+0000 7fa5c95eb740 1 -- start start 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.119+0000 7fa5c95eb740 1 Processor -- start 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94b7800 0x5562d94b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5562d94ba3d0 con 0x5562d94ac620 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562d93ac0b0 con 0x5562d94b7800 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5562d94ba550 con 0x5562d94acf40 2024-04-23T18:17:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c95eb740 10 client.0 _ll_register_callbacks cb 0x5562d9399d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5c4e6c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94b7800 0x5562d94b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5bf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5bf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39498/0 (socket says 172.21.15.32:39498) 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.120+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/4169136526 learned_addr learned my addr 172.21.15.32:0/4169136526 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bffff640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94b7800 msgr2=0x5562d94b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94b7800 0x5562d94b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/4169136526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 msgr2=0x5562d94b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 -- 172.21.15.32:0/4169136526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562d91f0620 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000c00 tx=0x7fa5b0056680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81299c2ba0f3b71b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.121+0000 7fa59f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0035050 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa5b0056a90 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b0051070 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa5b0056c90 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa5c95eb740 10 client.25465 fetch_fsmap learned FSMap version 28 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562d94ba3d0 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b0051070 con 0x5562d94acf40 2024-04-23T18:17:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa5b004d570 con 0x5562d94acf40 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.122+0000 7fa5c95eb740 10 client.25465 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5c95eb740 10 client.25465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5c95eb740 20 client.25465 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d94c77d0 0x5562d94c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5c95eb740 4 client.25465 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562d941f390 con 0x5562d94c77d0 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.123+0000 7fa5bffff640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d94c77d0 0x5562d94c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.124+0000 7fa5bffff640 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d94c77d0 0x5562d94c9c30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4583ad3779b4c672 server_cookie=2fc1d7d67f7bdbca in_seq=0 out_seq=0 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.124+0000 7fa59f7fe640 10 client.25465 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa59f7fe640 1 -- 172.21.15.32:0/4169136526 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562d941f390 con 0x5562d94c77d0 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa59f7fe640 10 client.25465 handle_command_reply: tid=0 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 client.25465 shutdown 2024-04-23T18:17:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 msgr2=0x5562d94b4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000c00 tx=0x7fa5b0056680 comp rx=0 tx=0).stop 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 shutdown_connections 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d94c77d0 0x5562d94c9c30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94b7800 0x5562d94b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94acf40 0x5562d94b4490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/4169136526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa59f7fe640 0 client.25465 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.125+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 >> 172.21.15.32:0/4169136526 conn(0x5562d94a8b30 msgr2=0x5562d94ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.126+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 shutdown_connections 2024-04-23T18:17:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.127+0000 7fa5c95eb740 1 -- 172.21.15.32:0/4169136526 wait complete. 2024-04-23T18:17:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.127+0000 7fa5c95eb740 20 client.25465 trim_cache size 0 max 16384 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 Processor -- start 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 -- start start 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94b7800 0x5562d94cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01b6ac0 con 0x5562d94b7800 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01b4320 con 0x5562d94acf40 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.154+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0050530 con 0x5562d94ac620 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39514/0 (socket says 172.21.15.32:39514) 2024-04-23T18:17:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 -- 172.21.15.32:0/3485983357 learned_addr learned my addr 172.21.15.32:0/3485983357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94b7800 0x5562d94cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 -- 172.21.15.32:0/3485983357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 msgr2=0x5562d94cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 -- 172.21.15.32:0/3485983357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94b7800 msgr2=0x5562d94cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94b7800 0x5562d94cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 -- 172.21.15.32:0/3485983357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562d94ad530 con 0x5562d94ac620 2024-04-23T18:17:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.155+0000 7fa5bffff640 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4098af0 tx=0x7fa5b40108d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9f7e6239c75e5f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.156+0000 7fa5a5ffb640 1 -- 172.21.15.32:0/3485983357 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40ba070 con 0x5562d94ac620 2024-04-23T18:17:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.156+0000 7fa5a5ffb640 1 -- 172.21.15.32:0/3485983357 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b409a420 con 0x5562d94ac620 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.156+0000 7fa5a5ffb640 1 -- 172.21.15.32:0/3485983357 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40ba070 con 0x5562d94ac620 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 -- 172.21.15.32:0/3485983357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 msgr2=0x5562d94c3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4098af0 tx=0x7fa5b40108d0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 -- 172.21.15.32:0/3485983357 shutdown_connections 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94b7800 0x5562d94cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/3485983357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94ac620 0x5562d94c3290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.157+0000 7fa5c95eb740 1 -- 172.21.15.32:0/3485983357 >> 172.21.15.32:0/3485983357 conn(0x5562d94a8b30 msgr2=0x5562d94a7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.158+0000 7fa5c95eb740 1 -- 172.21.15.32:0/3485983357 shutdown_connections 2024-04-23T18:17:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.158+0000 7fa5c95eb740 1 -- 172.21.15.32:0/3485983357 wait complete. 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.158+0000 7fa5c95eb740 1 -- start start 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 Processor -- start 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94bcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94bd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50858/0 (socket says 172.21.15.32:50858) 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0117b90 con 0x5562d94ac620 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0119000 con 0x5562d94acf40 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5c95eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0118260 con 0x5562d94b7800 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.159+0000 7fa5bffff640 1 -- 172.21.15.32:0/2836503766 learned_addr learned my addr 172.21.15.32:0/2836503766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94bd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c95eb740 10 client.0 _ll_register_callbacks cb 0x7fa5b0004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5bf7fe640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94bcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 -- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 msgr2=0x5562d94bcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94bcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 -- 172.21.15.32:0/2836503766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 msgr2=0x5562d94bf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 -- 172.21.15.32:0/2836503766 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562d94a83c0 con 0x5562d94b7800 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5bffff640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa5c4e6c640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94bd160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5562d94cff80 tx=0x7fa5b8004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16b6df1ea5097344 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa59f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.160+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b801dd50 con 0x5562d94b7800 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b409a420 con 0x5562d94b7800 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa5b801b450 con 0x5562d94b7800 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa5b8034050 con 0x5562d94b7800 2024-04-23T18:17:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 10 client.25471 fetch_fsmap learned FSMap version 28 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562d952d8f0 con 0x5562d94b7800 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b409a420 con 0x5562d94b7800 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa5b8026650 con 0x5562d94b7800 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 10 client.25471 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 10 client.25471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 20 client.25471 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d952e110 0x5562d9530530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 4 client.25471 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.161+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562d9419200 con 0x5562d952e110 2024-04-23T18:17:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.162+0000 7fa5bffff640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d952e110 0x5562d9530530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.163+0000 7fa5bffff640 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d952e110 0x5562d9530530 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd7dda08cd13bd8 server_cookie=7777292a028426ed in_seq=0 out_seq=0 2024-04-23T18:17:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.163+0000 7fa59f7fe640 10 client.25471 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa59f7fe640 1 -- 172.21.15.32:0/2836503766 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562d9419200 con 0x5562d952e110 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa59f7fe640 10 client.25471 handle_command_reply: tid=0 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 client.25471 shutdown 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 msgr2=0x5562d94bd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94bd160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5562d94cff80 tx=0x7fa5b8004ab0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 shutdown_connections 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5562d952e110 0x5562d9530530 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5562d94b7800 0x5562d94bd160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562d94acf40 0x5562d94bcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 --2- 172.21.15.32:0/2836503766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562d94ac620 0x5562d94bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa59f7fe640 0 client.25471 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 >> 172.21.15.32:0/2836503766 conn(0x5562d94a8b30 msgr2=0x5562d94c2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 shutdown_connections 2024-04-23T18:17:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.178+0000 7fa5c95eb740 1 -- 172.21.15.32:0/2836503766 wait complete. 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.179+0000 7fa5c95eb740 20 client.25471 trim_cache size 0 max 16384 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.180+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 msgr2=0x7fa5a80a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.180+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 0x7fa5a80a6360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8008e50 tx=0x7fa5b8004a40 comp rx=0 tx=0).stop 2024-04-23T18:17:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.180+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 msgr2=0x7fa5c01a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.180+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400b450 tx=0x7fa5b400b910 comp rx=0 tx=0).stop 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 shutdown_connections 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa5a80a3ea0 0x7fa5a80a6360 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5c011b730 0x7fa5c01abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5c01106f0 0x7fa5c01a7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 --2- 172.21.15.32:0/1179442006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c010e7d0 0x7fa5c01a75e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 >> 172.21.15.32:0/1179442006 conn(0x7fa5c006d030 msgr2=0x7fa5c011a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.181+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 shutdown_connections 2024-04-23T18:17:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:36.182+0000 7fa5c68cf640 1 -- 172.21.15.32:0/1179442006 wait complete. 2024-04-23T18:17:37.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:37 smithi032 ceph-mon[29221]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 730 op/s 2024-04-23T18:17:37.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:37 smithi040 ceph-mon[32626]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 730 op/s 2024-04-23T18:17:37.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:37 smithi064 ceph-mon[33062]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 730 op/s 2024-04-23T18:17:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 130.0 s, 3899.4 tps, lat 25.556 ms stddev 26.401 2024-04-23T18:17:39.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:39 smithi032 ceph-mon[29221]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-04-23T18:17:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:39 smithi040 ceph-mon[32626]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-04-23T18:17:39.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:39 smithi064 ceph-mon[33062]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-04-23T18:17:41.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:41 smithi032 ceph-mon[29221]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-23T18:17:41.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:41 smithi040 ceph-mon[32626]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-23T18:17:41.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:41 smithi064 ceph-mon[33062]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-23T18:17:43.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:43 smithi032 ceph-mon[29221]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.8 MiB/s wr, 660 op/s 2024-04-23T18:17:43.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:43 smithi040 ceph-mon[32626]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.8 MiB/s wr, 660 op/s 2024-04-23T18:17:43.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:43 smithi064 ceph-mon[33062]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.8 MiB/s wr, 660 op/s 2024-04-23T18:17:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 135.0 s, 3757.2 tps, lat 26.682 ms stddev 28.072 2024-04-23T18:17:45.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:45 smithi040 ceph-mon[32626]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-23T18:17:45.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:45 smithi064 ceph-mon[33062]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-23T18:17:45.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:45 smithi032 ceph-mon[29221]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-23T18:17:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:47 smithi040 ceph-mon[32626]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 745 op/s 2024-04-23T18:17:47.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:47 smithi064 ceph-mon[33062]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 745 op/s 2024-04-23T18:17:47.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:47 smithi032 ceph-mon[29221]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 745 op/s 2024-04-23T18:17:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 140.0 s, 3656.8 tps, lat 27.329 ms stddev 30.008 2024-04-23T18:17:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.322+0000 7f9951474640 1 -- 172.21.15.32:0/3842652412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c1061d0 msgr2=0x7f994c06a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.322+0000 7f9951474640 1 --2- 172.21.15.32:0/3842652412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c1061d0 0x7f994c06a160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9944009be0 tx=0x7f994402a6f0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 -- 172.21.15.32:0/3842652412 shutdown_connections 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 --2- 172.21.15.32:0/3842652412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c1061d0 0x7f994c06a160 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 --2- 172.21.15.32:0/3842652412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c101ce0 0x7f994c105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 --2- 172.21.15.32:0/3842652412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c101330 0x7f994c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 -- 172.21.15.32:0/3842652412 >> 172.21.15.32:0/3842652412 conn(0x7f994c077d30 msgr2=0x7f994c078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 -- 172.21.15.32:0/3842652412 shutdown_connections 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 -- 172.21.15.32:0/3842652412 wait complete. 2024-04-23T18:17:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.323+0000 7f9951474640 1 Processor -- start 2024-04-23T18:17:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 -- start start 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c101330 0x7f994c10d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c1061d0 0x7f994c10d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f994c110110 con 0x7f994c1a48f0 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f994c10ff90 con 0x7f994c1061d0 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f9951474640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f994c110290 con 0x7f994c101330 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f994b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f994b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55112/0 (socket says 172.21.15.32:55112) 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.324+0000 7f994b7fe640 1 -- 172.21.15.32:0/1600690813 learned_addr learned my addr 172.21.15.32:0/1600690813 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994a7fc640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c1061d0 0x7f994c10d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994affd640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c101330 0x7f994c10d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994b7fe640 1 -- 172.21.15.32:0/1600690813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c101330 msgr2=0x7f994c10d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994b7fe640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c101330 0x7f994c10d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994b7fe640 1 -- 172.21.15.32:0/1600690813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c1061d0 msgr2=0x7f994c10d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994b7fe640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c1061d0 0x7f994c10d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.325+0000 7f994b7fe640 1 -- 172.21.15.32:0/1600690813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994c1a5360 con 0x7f994c1a48f0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.326+0000 7f994b7fe640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9944003cf0 tx=0x7f9944003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.326+0000 7f9943fff640 1 -- 172.21.15.32:0/1600690813 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944040020 con 0x7f994c1a48f0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.326+0000 7f9943fff640 1 -- 172.21.15.32:0/1600690813 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9944003e40 con 0x7f994c1a48f0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.326+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994c1a55f0 con 0x7f994c1a48f0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.326+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f994c06bba0 con 0x7f994c1a48f0 2024-04-23T18:17:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.327+0000 7f9943fff640 1 -- 172.21.15.32:0/1600690813 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944036040 con 0x7f994c1a48f0 2024-04-23T18:17:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.330+0000 7f9954181740 1 Processor -- start 2024-04-23T18:17:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.330+0000 7f9943fff640 1 -- 172.21.15.32:0/1600690813 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9944043020 con 0x7f994c1a48f0 2024-04-23T18:17:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.330+0000 7f9954181740 1 -- start start 2024-04-23T18:17:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0df40 0x561f1ed182c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9943fff640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 0x7f99380a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:33348/0 (socket says 172.21.15.32:33348) 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f994a7fc640 1 -- 172.21.15.32:0/464164739 learned_addr learned my addr 172.21.15.32:0/464164739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561f1ec72970 con 0x561f1ed18800 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f1ec0f4a0 con 0x561f1ed0df40 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.331+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561f1ec6e930 con 0x561f1ed0d620 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994b7fe640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 0x7f99380a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994affd640 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0df40 0x561f1ed182c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f9943fff640 1 -- 172.21.15.32:0/1600690813 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f99440f1340 con 0x7f994c1a48f0 2024-04-23T18:17:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 -- 172.21.15.32:0/464164739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0df40 msgr2=0x561f1ed182c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0df40 0x561f1ed182c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 -- 172.21.15.32:0/464164739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 msgr2=0x561f1ed1abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.332+0000 7f994a7fc640 1 -- 172.21.15.32:0/464164739 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f1e985210 con 0x561f1ed0d620 2024-04-23T18:17:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.333+0000 7f994a7fc640 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9934009fa0 tx=0x7f9934009480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdfe771f22876447 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.333+0000 7f99417fa640 1 -- 172.21.15.32:0/464164739 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934017e60 con 0x561f1ed0d620 2024-04-23T18:17:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.333+0000 7f994b7fe640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 0x7f99380a62a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f99440b9790 tx=0x7f9944040fc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.333+0000 7f99417fa640 1 -- 172.21.15.32:0/464164739 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993401d070 con 0x561f1ed0d620 2024-04-23T18:17:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.333+0000 7f99417fa640 1 -- 172.21.15.32:0/464164739 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99340189e0 con 0x561f1ed0d620 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.334+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 msgr2=0x561f1ed0da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.334+0000 7f9954181740 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9934009fa0 tx=0x7f9934009480 comp rx=0 tx=0).stop 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.334+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 shutdown_connections 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.334+0000 7f9954181740 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.335+0000 7f9954181740 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0df40 0x561f1ed182c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.335+0000 7f9954181740 1 --2- 172.21.15.32:0/464164739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0d620 0x561f1ed0da00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.335+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 >> 172.21.15.32:0/464164739 conn(0x561f1ed09b30 msgr2=0x561f1ed09f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.336+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 shutdown_connections 2024-04-23T18:17:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.336+0000 7f9954181740 1 -- 172.21.15.32:0/464164739 wait complete. 2024-04-23T18:17:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 -- start start 2024-04-23T18:17:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 Processor -- start 2024-04-23T18:17:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed153e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed15920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561f1ed1b3d0 con 0x561f1ed18800 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f994affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f1ec0d0b0 con 0x561f1ed0d620 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561f1ed1b550 con 0x561f1ed0df40 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46340/0 (socket says 172.21.15.32:46340) 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f994a7fc640 1 -- 172.21.15.32:0/3952939626 learned_addr learned my addr 172.21.15.32:0/3952939626 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.337+0000 7f9954181740 10 client.0 _ll_register_callbacks cb 0x561f1ebfad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed15920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 -- 172.21.15.32:0/3952939626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 msgr2=0x561f1ed153e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed153e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 -- 172.21.15.32:0/3952939626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 msgr2=0x561f1ed15920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed15920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.338+0000 7f994a7fc640 1 -- 172.21.15.32:0/3952939626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f1ea51620 con 0x561f1ed0d620 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f994a7fc640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f99340186c0 tx=0x7f9934006a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4218096967ee48d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f99327fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934018d80 con 0x561f1ed0d620 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9934015730 con 0x561f1ed0d620 2024-04-23T18:17:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993401d070 con 0x561f1ed0d620 2024-04-23T18:17:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9934033050 con 0x561f1ed0d620 2024-04-23T18:17:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.339+0000 7f9954181740 10 client.25502 fetch_fsmap learned FSMap version 28 2024-04-23T18:17:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f1ed27320 con 0x561f1ed0d620 2024-04-23T18:17:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9934015de0 con 0x561f1ed0d620 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9934022790 con 0x561f1ed0d620 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 10 client.25502 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 10 client.25502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 20 client.25502 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed28780 0x561f1ed2ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 4 client.25502 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.340+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f993401d070 con 0x561f1ed28780 2024-04-23T18:17:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.342+0000 7f994affd640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed28780 0x561f1ed2ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.342+0000 7f994affd640 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed28780 0x561f1ed2ab70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7196a38cebe8b3 server_cookie=4c1becda4a4759ae in_seq=0 out_seq=0 2024-04-23T18:17:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.342+0000 7f99327fc640 10 client.25502 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.343+0000 7f99327fc640 1 -- 172.21.15.32:0/3952939626 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f993401d070 con 0x561f1ed28780 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.343+0000 7f99327fc640 10 client.25502 handle_command_reply: tid=0 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.343+0000 7f9954181740 1 client.25502 shutdown 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 msgr2=0x561f1ed180f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f99340186c0 tx=0x7f9934006a30 comp rx=0 tx=0).stop 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 shutdown_connections 2024-04-23T18:17:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed28780 0x561f1ed2ab70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed15920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f99327fc640 0 client.25502 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 --2- 172.21.15.32:0/3952939626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed180f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 >> 172.21.15.32:0/3952939626 conn(0x561f1ed09b30 msgr2=0x561f1ed0f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 shutdown_connections 2024-04-23T18:17:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.344+0000 7f9954181740 1 -- 172.21.15.32:0/3952939626 wait complete. 2024-04-23T18:17:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.345+0000 7f9954181740 20 client.25502 trim_cache size 0 max 16384 2024-04-23T18:17:49.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:49 smithi040 ceph-mon[32626]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.374+0000 7f9954181740 1 Processor -- start 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 -- start start 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 0x561f1ed242f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed18800 0x561f1ed30290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f992c0110c0 con 0x561f1ed0d620 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f992c00e8d0 con 0x561f1ed18800 2024-04-23T18:17:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.375+0000 7f9954181740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f992c00c720 con 0x561f1ed0df40 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 0x561f1ed242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:33354/0 (socket says 172.21.15.32:33354) 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 -- 172.21.15.32:0/1404411962 learned_addr learned my addr 172.21.15.32:0/1404411962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994a7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 0x561f1ed242f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55130/0 (socket says 172.21.15.32:55130) 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994a7fc640 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed18800 0x561f1ed30290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 -- 172.21.15.32:0/1404411962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed18800 msgr2=0x561f1ed30290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed18800 0x561f1ed30290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 -- 172.21.15.32:0/1404411962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 msgr2=0x561f1ed242f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 0x561f1ed242f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 -- 172.21.15.32:0/1404411962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f1ed092d0 con 0x561f1ed0df40 2024-04-23T18:17:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f994affd640 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x561f1ed0ebc0 tx=0x7f993c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30c1b26b367f599 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.376+0000 7f9940ff9640 1 -- 172.21.15.32:0/1404411962 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c027070 con 0x561f1ed0df40 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.377+0000 7f9940ff9640 1 -- 172.21.15.32:0/1404411962 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c004230 con 0x561f1ed0df40 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.377+0000 7f9940ff9640 1 -- 172.21.15.32:0/1404411962 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c003950 con 0x561f1ed0df40 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- 172.21.15.32:0/1404411962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 msgr2=0x561f1ed08420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x561f1ed0ebc0 tx=0x7f993c009260 comp rx=0 tx=0).stop 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- 172.21.15.32:0/1404411962 shutdown_connections 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed18800 0x561f1ed30290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed08420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 --2- 172.21.15.32:0/1404411962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed0d620 0x561f1ed242f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- 172.21.15.32:0/1404411962 >> 172.21.15.32:0/1404411962 conn(0x561f1ed09b30 msgr2=0x561f1ed15500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- 172.21.15.32:0/1404411962 shutdown_connections 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- 172.21.15.32:0/1404411962 wait complete. 2024-04-23T18:17:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 -- start start 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.378+0000 7f9954181740 1 Processor -- start 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed20b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f992c00cd90 con 0x561f1ed18800 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f992c00e200 con 0x561f1ed0d620 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f992c00d460 con 0x561f1ed0df40 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f9954181740 10 client.0 _ll_register_callbacks cb 0x7f9934017f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f994affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f994affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:33362/0 (socket says 172.21.15.32:33362) 2024-04-23T18:17:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.379+0000 7f994affd640 1 -- 172.21.15.32:0/1042636684 learned_addr learned my addr 172.21.15.32:0/1042636684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 -- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 msgr2=0x561f1ed20b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 -- 172.21.15.32:0/1042636684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 msgr2=0x561f1ed1e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 -- 172.21.15.32:0/1042636684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f1ed1ebd0 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f994affd640 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f993c009640 tx=0x7f993c0043e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d26c309230457ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.380+0000 7f99327fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c030020 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f993c021360 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993c004230 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f993c03b050 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f9954181740 10 client.25489 fetch_fsmap learned FSMap version 28 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.381+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f1ed8ea90 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c004230 con 0x561f1ed0df40 2024-04-23T18:17:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f993c003da0 con 0x561f1ed0df40 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 10 client.25489 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 10 client.25489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 20 client.25489 populate_metadata read hostname 'smithi032' 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed8f240 0x561f1ed91660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 4 client.25489 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.382+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f1ec80390 con 0x561f1ed8f240 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.383+0000 7f994a7fc640 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed8f240 0x561f1ed91660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.383+0000 7f994a7fc640 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed8f240 0x561f1ed91660 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c486dc8651d7aacf server_cookie=1195bed435eff8f8 in_seq=0 out_seq=0 2024-04-23T18:17:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.383+0000 7f99327fc640 10 client.25489 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.394+0000 7f99327fc640 1 -- 172.21.15.32:0/1042636684 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f1ec80390 con 0x561f1ed8f240 2024-04-23T18:17:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.395+0000 7f99327fc640 10 client.25489 handle_command_reply: tid=0 2024-04-23T18:17:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.395+0000 7f9954181740 1 client.25489 shutdown 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.395+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 msgr2=0x561f1ed1de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.395+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f993c009640 tx=0x7f993c0043e0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 shutdown_connections 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561f1ed8f240 0x561f1ed91660 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f1ed18800 0x561f1ed1e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561f1ed0df40 0x561f1ed1de90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 --2- 172.21.15.32:0/1042636684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f1ed0d620 0x561f1ed20b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 >> 172.21.15.32:0/1042636684 conn(0x561f1ed09b30 msgr2=0x561f1ed232e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f99327fc640 0 client.25489 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 shutdown_connections 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.396+0000 7f9954181740 1 -- 172.21.15.32:0/1042636684 wait complete. 2024-04-23T18:17:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.397+0000 7f9954181740 20 client.25489 trim_cache size 0 max 16384 2024-04-23T18:17:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:17:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:17:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:17:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 msgr2=0x7f99380a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 0x7f99380a62a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f99440b9790 tx=0x7f9944040fc0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 msgr2=0x7f994c1a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9944003cf0 tx=0x7f9944003d20 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 shutdown_connections 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99380a3de0 0x7f99380a62a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994c1a48f0 0x7f994c1a4da0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f994c1061d0 0x7f994c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 --2- 172.21.15.32:0/1600690813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f994c101330 0x7f994c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.399+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 >> 172.21.15.32:0/1600690813 conn(0x7f994c077d30 msgr2=0x7f994c111970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.400+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 shutdown_connections 2024-04-23T18:17:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:17:49.400+0000 7f9951474640 1 -- 172.21.15.32:0/1600690813 wait complete. 2024-04-23T18:17:49.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:49 smithi064 ceph-mon[33062]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-04-23T18:17:49.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:49 smithi032 ceph-mon[29221]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-04-23T18:17:51.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:51 smithi040 ceph-mon[32626]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-23T18:17:51.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:51 smithi064 ceph-mon[33062]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-23T18:17:51.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:51 smithi032 ceph-mon[29221]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-23T18:17:53.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:53 smithi040 ceph-mon[32626]: pgmap v86: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 692 op/s 2024-04-23T18:17:53.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:53 smithi064 ceph-mon[33062]: pgmap v86: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 692 op/s 2024-04-23T18:17:53.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:53 smithi032 ceph-mon[29221]: pgmap v86: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 692 op/s 2024-04-23T18:17:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 145.0 s, 3554.0 tps, lat 28.164 ms stddev 29.172 2024-04-23T18:17:55.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:55 smithi040 ceph-mon[32626]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:17:55.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:55 smithi064 ceph-mon[33062]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:17:55.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:55 smithi032 ceph-mon[29221]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:17:57.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:57 smithi040 ceph-mon[32626]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:17:57.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:57 smithi064 ceph-mon[33062]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:17:57.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:57 smithi032 ceph-mon[29221]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:17:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 150.0 s, 3523.8 tps, lat 28.189 ms stddev 29.544 2024-04-23T18:17:59.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:17:59 smithi040 ceph-mon[32626]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:17:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:17:59 smithi064 ceph-mon[33062]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:17:59.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:17:59 smithi032 ceph-mon[29221]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 MiB/s wr, 635 op/s 2024-04-23T18:18:01.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:01 smithi040 ceph-mon[32626]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 949 op/s 2024-04-23T18:18:01.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:01 smithi064 ceph-mon[33062]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 949 op/s 2024-04-23T18:18:01.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:01 smithi032 ceph-mon[29221]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 949 op/s 2024-04-23T18:18:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 -- 172.21.15.32:0/1408207635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c1098f0 msgr2=0x7f1d3c07a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 --2- 172.21.15.32:0/1408207635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c07a1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009ca0 tx=0x7f1d3402a8d0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 -- 172.21.15.32:0/1408207635 shutdown_connections 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 --2- 172.21.15.32:0/1408207635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c07a1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 --2- 172.21.15.32:0/1408207635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c108f50 0x7f1d3c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 --2- 172.21.15.32:0/1408207635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c10e600 0x7f1d3c10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 -- 172.21.15.32:0/1408207635 >> 172.21.15.32:0/1408207635 conn(0x7f1d3c06d030 msgr2=0x7f1d3c06d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.559+0000 7f1d43620640 1 -- 172.21.15.32:0/1408207635 shutdown_connections 2024-04-23T18:18:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.560+0000 7f1d43620640 1 -- 172.21.15.32:0/1408207635 wait complete. 2024-04-23T18:18:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.561+0000 7f1d43620640 1 Processor -- start 2024-04-23T18:18:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 -- start start 2024-04-23T18:18:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c074950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 0x7f1d3c074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c07f880 con 0x7f1d3c108f50 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c07f700 con 0x7f1d3c10e600 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.562+0000 7f1d43620640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c07fa00 con 0x7f1d3c1098f0 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51316/0 (socket says 172.21.15.32:51316) 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 -- 172.21.15.32:0/966415329 learned_addr learned my addr 172.21.15.32:0/966415329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 0x7f1d3c074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c074950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 -- 172.21.15.32:0/966415329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c1098f0 msgr2=0x7f1d3c074950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c074950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 -- 172.21.15.32:0/966415329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 msgr2=0x7f1d3c074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 0x7f1d3c074e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d413bc640 1 -- 172.21.15.32:0/966415329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c1bc180 con 0x7f1d3c108f50 2024-04-23T18:18:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.563+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 0x7f1d3c074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.564+0000 7f1d413bc640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800b4a0 tx=0x7f1d3800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.564+0000 7f1d327fc640 1 -- 172.21.15.32:0/966415329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38011830 con 0x7f1d3c108f50 2024-04-23T18:18:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.565+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c1bc470 con 0x7f1d3c108f50 2024-04-23T18:18:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.565+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c1bc9c0 con 0x7f1d3c108f50 2024-04-23T18:18:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.566+0000 7f1d327fc640 1 -- 172.21.15.32:0/966415329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d38016070 con 0x7f1d3c108f50 2024-04-23T18:18:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.566+0000 7f1d327fc640 1 -- 172.21.15.32:0/966415329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38010e10 con 0x7f1d3c108f50 2024-04-23T18:18:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.569+0000 7f1d4640c740 1 Processor -- start 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.571+0000 7f1d4640c740 1 -- start start 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.571+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.571+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 0x55ff95ccf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.571+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95cd1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.572+0000 7f1d4640c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ff95c29970 con 0x55ff95ccf800 2024-04-23T18:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.572+0000 7f1d4640c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ff95bc64a0 con 0x55ff95cc4f40 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.572+0000 7f1d4640c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ff95c25930 con 0x55ff95cc4620 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.573+0000 7f1d327fc640 1 -- 172.21.15.32:0/966415329 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1d3801b4a0 con 0x7f1d3c108f50 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.573+0000 7f1d41bbd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.573+0000 7f1d41bbd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:38164/0 (socket says 172.21.15.32:38164) 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.573+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1300660619 learned_addr learned my addr 172.21.15.32:0/1300660619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.573+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 0x55ff95ccf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.574+0000 7f1d327fc640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 0x7f1d280a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.575+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95cd1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.575+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 0x7f1d280a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1300660619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 msgr2=0x55ff95ccf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 0x55ff95ccf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1300660619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 msgr2=0x55ff95cd1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95cd1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1300660619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff9593c210 con 0x55ff95cc4620 2024-04-23T18:18:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 0x55ff95ccf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.577+0000 7f1d327fc640 1 -- 172.21.15.32:0/966415329 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1d380cf9f0 con 0x7f1d3c108f50 2024-04-23T18:18:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.580+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009ca0 tx=0x7f1d34031760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=295f6850c7f60681 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.581+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 0x7f1d280a63e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1d34035000 tx=0x7f1d3404a000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.581+0000 7f1d277fe640 1 -- 172.21.15.32:0/1300660619 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d34004040 con 0x55ff95cc4620 2024-04-23T18:18:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.581+0000 7f1d277fe640 1 -- 172.21.15.32:0/1300660619 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d34036040 con 0x55ff95cc4620 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 -- 172.21.15.32:0/1300660619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 msgr2=0x55ff95cc4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009ca0 tx=0x7f1d34031760 comp rx=0 tx=0).stop 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 -- 172.21.15.32:0/1300660619 shutdown_connections 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95cd1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4f40 0x55ff95ccf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1300660619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4620 0x55ff95cc4a00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.582+0000 7f1d4640c740 1 -- 172.21.15.32:0/1300660619 >> 172.21.15.32:0/1300660619 conn(0x55ff95cc0b30 msgr2=0x55ff95cc0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.583+0000 7f1d4640c740 1 -- 172.21.15.32:0/1300660619 shutdown_connections 2024-04-23T18:18:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.584+0000 7f1d4640c740 1 -- 172.21.15.32:0/1300660619 wait complete. 2024-04-23T18:18:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 -- start start 2024-04-23T18:18:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 Processor -- start 2024-04-23T18:18:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95ccc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95ccc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ff95cd23d0 con 0x55ff95cc4620 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ff95bc40b0 con 0x55ff95ccf800 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ff95cd2550 con 0x55ff95cc4f40 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.585+0000 7f1d4640c740 10 client.0 _ll_register_callbacks cb 0x55ff95bb1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51332/0 (socket says 172.21.15.32:51332) 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 -- 172.21.15.32:0/1004506106 learned_addr learned my addr 172.21.15.32:0/1004506106 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95ccc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 -- 172.21.15.32:0/1004506106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 msgr2=0x55ff95ccc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95ccc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 -- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 msgr2=0x55ff95ccc8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95ccc8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.586+0000 7f1d413bc640 1 -- 172.21.15.32:0/1004506106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff95a08620 con 0x55ff95cc4620 2024-04-23T18:18:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.587+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1d38098460 tx=0x7f1d3800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5492c8110ecfe543 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.590+0000 7f1d24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.590+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3809ba90 con 0x55ff95cc4620 2024-04-23T18:18:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.590+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d34036040 con 0x55ff95cc4620 2024-04-23T18:18:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.590+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1d380cd3b0 con 0x55ff95cc4620 2024-04-23T18:18:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.593+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1d3809fe40 con 0x55ff95cc4620 2024-04-23T18:18:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.595+0000 7f1d4640c740 10 client.15864 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.595+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ff95cde330 con 0x55ff95cc4620 2024-04-23T18:18:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.596+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d34036040 con 0x55ff95cc4620 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.598+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1d380a4300 con 0x55ff95cc4620 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.598+0000 7f1d4640c740 10 client.15864 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.598+0000 7f1d4640c740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.598+0000 7f1d4640c740 20 client.15864 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.598+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95cdfca0 0x55ff95ce2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.599+0000 7f1d4640c740 4 client.15864 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.599+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff95c37390 con 0x55ff95cdfca0 2024-04-23T18:18:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.599+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95cdfca0 0x55ff95ce2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.601+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95cdfca0 0x55ff95ce2120 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=999deb48370e8593 server_cookie=2b569314544ab12e in_seq=0 out_seq=0 2024-04-23T18:18:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.602+0000 7f1d24ff9640 10 client.15864 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.602+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1004506106 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff95c37390 con 0x55ff95cdfca0 2024-04-23T18:18:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.602+0000 7f1d24ff9640 10 client.15864 handle_command_reply: tid=0 2024-04-23T18:18:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.603+0000 7f1d4640c740 1 client.15864 shutdown 2024-04-23T18:18:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 msgr2=0x55ff95ccf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1d38098460 tx=0x7f1d3800bf90 comp rx=0 tx=0).stop 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 shutdown_connections 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95cdfca0 0x55ff95ce2120 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95ccc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95ccc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1004506106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95ccf090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d24ff9640 0 client.15864 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 >> 172.21.15.32:0/1004506106 conn(0x55ff95cc0b30 msgr2=0x55ff95cc6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 shutdown_connections 2024-04-23T18:18:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 1 -- 172.21.15.32:0/1004506106 wait complete. 2024-04-23T18:18:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.604+0000 7f1d4640c740 20 client.15864 trim_cache size 0 max 16384 2024-04-23T18:18:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.638+0000 7f1d4640c740 1 Processor -- start 2024-04-23T18:18:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.638+0000 7f1d4640c740 1 -- start start 2024-04-23T18:18:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cc7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95ce7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d4640c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20011050 con 0x55ff95ccf800 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:50488/0 (socket says 172.21.15.32:50488) 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d413bc640 1 -- 172.21.15.32:0/2299010747 learned_addr learned my addr 172.21.15.32:0/2299010747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cc7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95ce7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.639+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000e860 con 0x55ff95cc4620 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000c6b0 con 0x55ff95cc4f40 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d413bc640 1 -- 172.21.15.32:0/2299010747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 msgr2=0x55ff95cc7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d413bc640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cc7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d413bc640 1 -- 172.21.15.32:0/2299010747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 msgr2=0x55ff95ce7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d413bc640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95ce7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d413bc640 1 -- 172.21.15.32:0/2299010747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff95cbc2e0 con 0x55ff95cc4620 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.640+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cc7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.641+0000 7f1d413bc640 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1d38097950 tx=0x7f1d380b5140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8032006aa4d4ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.641+0000 7f1d26ffd640 1 -- 172.21.15.32:0/2299010747 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38097b50 con 0x55ff95cc4620 2024-04-23T18:18:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.641+0000 7f1d26ffd640 1 -- 172.21.15.32:0/2299010747 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d3809ba90 con 0x55ff95cc4620 2024-04-23T18:18:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.641+0000 7f1d26ffd640 1 -- 172.21.15.32:0/2299010747 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d380a4270 con 0x55ff95cc4620 2024-04-23T18:18:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 msgr2=0x55ff95cdb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1d38097950 tx=0x7f1d380b5140 comp rx=0 tx=0).stop 2024-04-23T18:18:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 shutdown_connections 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95ccf800 0x55ff95ce7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cc7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 --2- 172.21.15.32:0/2299010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95cc4620 0x55ff95cdb370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.642+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 >> 172.21.15.32:0/2299010747 conn(0x55ff95cc0b30 msgr2=0x55ff95cc7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.643+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 shutdown_connections 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.643+0000 7f1d4640c740 1 -- 172.21.15.32:0/2299010747 wait complete. 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.644+0000 7f1d4640c740 1 -- start start 2024-04-23T18:18:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.644+0000 7f1d4640c740 1 Processor -- start 2024-04-23T18:18:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95cd7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d4640c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cd7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95cd7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d413bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95cd7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51352/0 (socket says 172.21.15.32:51352) 2024-04-23T18:18:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d413bc640 1 -- 172.21.15.32:0/1418897850 learned_addr learned my addr 172.21.15.32:0/1418897850 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d40bbb640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cd7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.645+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95cd2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000cd20 con 0x55ff95cc4620 2024-04-23T18:18:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000e190 con 0x55ff95ccf800 2024-04-23T18:18:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000d3f0 con 0x55ff95cc4f40 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95cd2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d4640c740 10 client.0 _ll_register_callbacks cb 0x7f1d38097dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1418897850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 msgr2=0x55ff95cd7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cd7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1418897850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 msgr2=0x55ff95cd7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95cd7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.646+0000 7f1d41bbd640 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff95cd3730 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d41bbd640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95cd2f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d34031730 tx=0x7f1d3402fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f5280b165b8d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3405a070 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1d340305d0 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d3809ba90 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.647+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1d340098d0 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.648+0000 7f1d4640c740 10 client.25520 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.648+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ff95d45ef0 con 0x55ff95ccf800 2024-04-23T18:18:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.648+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d34030c80 con 0x55ff95ccf800 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.648+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1d34055070 con 0x55ff95ccf800 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 10 client.25520 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 10 client.25520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 20 client.25520 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95d46280 0x55ff95d48670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 4 client.25520 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.649+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f1d3809ba90 con 0x55ff95d46280 2024-04-23T18:18:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.650+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95d46280 0x55ff95d48670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.651+0000 7f1d413bc640 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95d46280 0x55ff95d48670 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76c2a7846ffefbdd server_cookie=994f62cff08568ed in_seq=0 out_seq=0 2024-04-23T18:18:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.651+0000 7f1d24ff9640 10 client.25520 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:02.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d24ff9640 1 -- 172.21.15.32:0/1418897850 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f1d3809ba90 con 0x55ff95d46280 2024-04-23T18:18:02.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d24ff9640 10 client.25520 handle_command_reply: tid=0 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 client.25520 shutdown 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 msgr2=0x55ff95cd2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95cd2f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d34031730 tx=0x7f1d3402fb20 comp rx=0 tx=0).stop 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 shutdown_connections 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55ff95d46280 0x55ff95d48670 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff95ccf800 0x55ff95cd2f00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff95cc4f40 0x55ff95cd7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 --2- 172.21.15.32:0/1418897850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff95cc4620 0x55ff95cd7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 >> 172.21.15.32:0/1418897850 conn(0x55ff95cc0b30 msgr2=0x55ff95cbf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.682+0000 7f1d24ff9640 0 client.25520 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.683+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 shutdown_connections 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.683+0000 7f1d4640c740 1 -- 172.21.15.32:0/1418897850 wait complete. 2024-04-23T18:18:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.683+0000 7f1d4640c740 20 client.25520 trim_cache size 0 max 16384 2024-04-23T18:18:02.678 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:18:02.678 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:18:02.678 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:18:02.678 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 msgr2=0x7f1d280a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 0x7f1d280a63e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1d34035000 tx=0x7f1d3404a000 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 msgr2=0x7f1d3c074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800b4a0 tx=0x7f1d3800b960 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 shutdown_connections 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d3c10e600 0x7f1d3c074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d3c1098f0 0x7f1d3c074950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1d280a3f20 0x7f1d280a63e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 --2- 172.21.15.32:0/966415329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d3c108f50 0x7f1d3c074410 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.679 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 >> 172.21.15.32:0/966415329 conn(0x7f1d3c06d030 msgr2=0x7f1d3c072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.680 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 shutdown_connections 2024-04-23T18:18:02.680 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:02.684+0000 7f1d43620640 1 -- 172.21.15.32:0/966415329 wait complete. 2024-04-23T18:18:03.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:03 smithi040 ceph-mon[32626]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 632 op/s 2024-04-23T18:18:03.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:03 smithi064 ceph-mon[33062]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 632 op/s 2024-04-23T18:18:03.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:03 smithi032 ceph-mon[29221]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 632 op/s 2024-04-23T18:18:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 155.0 s, 1652.0 tps, lat 60.849 ms stddev 157.893 2024-04-23T18:18:05.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:05 smithi040 ceph-mon[32626]: pgmap v92: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 1.70k op/s 2024-04-23T18:18:05.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:05 smithi064 ceph-mon[33062]: pgmap v92: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 1.70k op/s 2024-04-23T18:18:05.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:05 smithi032 ceph-mon[29221]: pgmap v92: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 1.70k op/s 2024-04-23T18:18:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:07 smithi040 ceph-mon[32626]: pgmap v93: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:07 smithi064 ceph-mon[33062]: pgmap v93: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:07.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:07 smithi032 ceph-mon[29221]: pgmap v93: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 160.0 s, 3691.6 tps, lat 27.139 ms stddev 28.592 2024-04-23T18:18:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:09 smithi040 ceph-mon[32626]: pgmap v94: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:09.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:09 smithi064 ceph-mon[33062]: pgmap v94: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:09.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:09 smithi032 ceph-mon[29221]: pgmap v94: 97 pgs: 97 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:11 smithi040 ceph-mon[32626]: pgmap v95: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 42 MiB/s wr, 1.70k op/s 2024-04-23T18:18:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:11 smithi064 ceph-mon[33062]: pgmap v95: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 42 MiB/s wr, 1.70k op/s 2024-04-23T18:18:11.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:11 smithi032 ceph-mon[29221]: pgmap v95: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 42 MiB/s wr, 1.70k op/s 2024-04-23T18:18:13.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:13 smithi032 ceph-mon[29221]: pgmap v96: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:13.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:13 smithi040 ceph-mon[32626]: pgmap v96: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:13.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:13 smithi064 ceph-mon[33062]: pgmap v96: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 39 MiB/s wr, 1.39k op/s 2024-04-23T18:18:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 165.0 s, 3551.8 tps, lat 28.105 ms stddev 30.389 2024-04-23T18:18:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:15 smithi040 ceph-mon[32626]: pgmap v97: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 1.85k op/s 2024-04-23T18:18:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:15 smithi064 ceph-mon[33062]: pgmap v97: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 1.85k op/s 2024-04-23T18:18:15.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:15 smithi032 ceph-mon[29221]: pgmap v97: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 1.85k op/s 2024-04-23T18:18:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/197156734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 msgr2=0x7f9df4102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/197156734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df4102590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009900 tx=0x7f9de802a400 comp rx=0 tx=0).stop 2024-04-23T18:18:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/197156734 shutdown_connections 2024-04-23T18:18:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/197156734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/197156734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df4102b60 0x7f9df4106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/197156734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df4102590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/197156734 >> 172.21.15.32:0/197156734 conn(0x7f9df40fda30 msgr2=0x7f9df40ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/197156734 shutdown_connections 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.824+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/197156734 wait complete. 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.825+0000 7f9dfaaa7640 1 Processor -- start 2024-04-23T18:18:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.825+0000 7f9dfaaa7640 1 -- start start 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.825+0000 7f9dfaaa7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.825+0000 7f9dfaaa7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df8843640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df8843640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:41168/0 (socket says 172.21.15.32:41168) 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9dfaaa7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df41a3650 0x7f9df41a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9dfaaa7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4110020 con 0x7f9df4107280 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54600/0 (socket says 172.21.15.32:54600) 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df3fff640 1 -- 172.21.15.32:0/4259230015 learned_addr learned my addr 172.21.15.32:0/4259230015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9df410fea0 con 0x7f9df41021b0 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41101a0 con 0x7f9df41a3650 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.826+0000 7f9df9044640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df41a3650 0x7f9df41a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 -- 172.21.15.32:0/4259230015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df41a3650 msgr2=0x7f9df41a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df41a3650 0x7f9df41a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 -- 172.21.15.32:0/4259230015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 msgr2=0x7f9df419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 -- 172.21.15.32:0/4259230015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df41a4140 con 0x7f9df4107280 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df8843640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c970 tx=0x7f9de000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df1ffb640 1 -- 172.21.15.32:0/4259230015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0007bb0 con 0x7f9df4107280 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9df1ffb640 1 -- 172.21.15.32:0/4259230015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de0007d50 con 0x7f9df4107280 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.827+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df41a4430 con 0x7f9df4107280 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.828+0000 7f9df1ffb640 1 -- 172.21.15.32:0/4259230015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de000f450 con 0x7f9df4107280 2024-04-23T18:18:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.828+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df41abc20 con 0x7f9df4107280 2024-04-23T18:18:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.832+0000 7f9df1ffb640 1 -- 172.21.15.32:0/4259230015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9de0004f20 con 0x7f9df4107280 2024-04-23T18:18:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.832+0000 7f9dfd7c1740 1 Processor -- start 2024-04-23T18:18:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.832+0000 7f9dfd7c1740 1 -- start start 2024-04-23T18:18:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf542c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf56bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564ffceae970 con 0x564ffcf49620 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564ffce4b4a0 con 0x564ffcf54800 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564ffceaa930 con 0x564ffcf49f40 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df1ffb640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 0x7f9de40a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df9044640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df9044640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54612/0 (socket says 172.21.15.32:54612) 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df9044640 1 -- 172.21.15.32:0/407827584 learned_addr learned my addr 172.21.15.32:0/407827584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df3fff640 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf56bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.833+0000 7f9df8843640 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df1ffb640 1 -- 172.21.15.32:0/4259230015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9de00d03d0 con 0x7f9df4107280 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 -- 172.21.15.32:0/407827584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 msgr2=0x564ffcf542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 -- 172.21.15.32:0/407827584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 msgr2=0x564ffcf56bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf56bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 -- 172.21.15.32:0/407827584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ffcbc1210 con 0x564ffcf49620 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 0x7f9de40a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.834+0000 7f9df9044640 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009fd0 tx=0x7f9dec0128f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce142609e14d80c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.835+0000 7f9ddeffd640 1 -- 172.21.15.32:0/407827584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec022070 con 0x564ffcf49620 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.835+0000 7f9ddeffd640 1 -- 172.21.15.32:0/407827584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dec013d70 con 0x564ffcf49620 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.835+0000 7f9ddeffd640 1 -- 172.21.15.32:0/407827584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec01a360 con 0x564ffcf49620 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/407827584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 msgr2=0x564ffcf49a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009fd0 tx=0x7f9dec0128f0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9df9044640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 0x7f9de40a6400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dec01bfd0 tx=0x7f9dec01a500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/407827584 shutdown_connections 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf56bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/407827584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf49a00 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.836+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/407827584 >> 172.21.15.32:0/407827584 conn(0x564ffcf45b30 msgr2=0x564ffcf45f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.837+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/407827584 shutdown_connections 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.837+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/407827584 wait complete. 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.837+0000 7f9dfd7c1740 1 -- start start 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 Processor -- start 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf51470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564ffcf57430 con 0x564ffcf49620 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564ffce490b0 con 0x564ffcf54800 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564ffcf575b0 con 0x564ffcf49f40 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9dfd7c1740 10 client.0 _ll_register_callbacks cb 0x564ffce36d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9df8843640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9df8843640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54622/0 (socket says 172.21.15.32:54622) 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.838+0000 7f9df8843640 1 -- 172.21.15.32:0/2593932788 learned_addr learned my addr 172.21.15.32:0/2593932788 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df3fff640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf51470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df9044640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 -- 172.21.15.32:0/2593932788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 msgr2=0x564ffcf51470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf51470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 -- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 msgr2=0x564ffcf519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 -- 172.21.15.32:0/2593932788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ffcc8d620 con 0x564ffcf49620 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9df8843640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9de802a970 tx=0x7f9de80053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212002d174b268b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9dd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8035040 con 0x564ffcf49620 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dec013d70 con 0x564ffcf49620 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.839+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9de80048c0 con 0x564ffcf49620 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9de8004ac0 con 0x564ffcf49620 2024-04-23T18:18:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 10 client.15882 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564ffcf63380 con 0x564ffcf49620 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dec013d70 con 0x564ffcf49620 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9de80314a0 con 0x564ffcf49620 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 10 client.15882 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 20 client.15882 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcf647f0 0x564ffcf66c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 4 client.15882 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ffcebc390 con 0x564ffcf647f0 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.840+0000 7f9df3fff640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcf647f0 0x564ffcf66c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.841+0000 7f9dd3fff640 10 client.15882 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.841+0000 7f9df3fff640 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcf647f0 0x564ffcf66c60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a928e4aed2f48a7 server_cookie=c41e9890d8ed64ed in_seq=0 out_seq=0 2024-04-23T18:18:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.843+0000 7f9dd3fff640 1 -- 172.21.15.32:0/2593932788 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ffcebc390 con 0x564ffcf647f0 2024-04-23T18:18:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.843+0000 7f9dd3fff640 10 client.15882 handle_command_reply: tid=0 2024-04-23T18:18:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.843+0000 7f9dfd7c1740 1 client.15882 shutdown 2024-04-23T18:18:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.844+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 msgr2=0x564ffcf54180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.844+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9de802a970 tx=0x7f9de80053d0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.839 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 shutdown_connections 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcf647f0 0x564ffcf66c60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf54800 0x564ffcf519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf51470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/2593932788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf49620 0x564ffcf54180 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 >> 172.21.15.32:0/2593932788 conn(0x564ffcf45b30 msgr2=0x564ffcf4b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.845+0000 7f9dd3fff640 0 client.15882 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.846+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 shutdown_connections 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.846+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/2593932788 wait complete. 2024-04-23T18:18:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.846+0000 7f9dfd7c1740 20 client.15882 trim_cache size 0 max 16384 2024-04-23T18:18:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 Processor -- start 2024-04-23T18:18:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 -- start start 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf53cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df8843640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df8843640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:44670/0 (socket says 172.21.15.32:44670) 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:41190/0 (socket says 172.21.15.32:41190) 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df9044640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf53cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9df8843640 1 -- 172.21.15.32:0/4260947380 learned_addr learned my addr 172.21.15.32:0/4260947380 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.870+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b6a80 con 0x564ffcf54800 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b4290 con 0x564ffcf49f40 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b20e0 con 0x564ffcf49620 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df8843640 1 -- 172.21.15.32:0/4260947380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 msgr2=0x564ffcf44b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df8843640 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df8843640 1 -- 172.21.15.32:0/4260947380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 msgr2=0x564ffcf53cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df8843640 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf53cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df8843640 1 -- 172.21.15.32:0/4260947380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ffcf49d70 con 0x564ffcf49620 2024-04-23T18:18:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.871+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.872+0000 7f9df8843640 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009850 tx=0x7f9de8031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7395cec912205fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.874+0000 7f9dde7fc640 1 -- 172.21.15.32:0/4260947380 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8035040 con 0x564ffcf49620 2024-04-23T18:18:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.874+0000 7f9dde7fc640 1 -- 172.21.15.32:0/4260947380 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de8040070 con 0x564ffcf49620 2024-04-23T18:18:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.874+0000 7f9dde7fc640 1 -- 172.21.15.32:0/4260947380 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8004920 con 0x564ffcf49620 2024-04-23T18:18:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 msgr2=0x564ffcf602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009850 tx=0x7f9de8031630 comp rx=0 tx=0).stop 2024-04-23T18:18:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 shutdown_connections 2024-04-23T18:18:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf53cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49f40 0x564ffcf44b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4260947380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49620 0x564ffcf602f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.875+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 >> 172.21.15.32:0/4260947380 conn(0x564ffcf45b30 msgr2=0x564ffcf44420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.876+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 shutdown_connections 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.876+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4260947380 wait complete. 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.876+0000 7f9dfd7c1740 1 -- start start 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.876+0000 7f9dfd7c1740 1 Processor -- start 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 0x564ffcf5ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf58650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b2750 con 0x564ffcf54800 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b3bc0 con 0x564ffcf49620 2024-04-23T18:18:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41b2e20 con 0x564ffcf49f40 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9dfd7c1740 10 client.0 _ll_register_callbacks cb 0x7f9de8004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 0x564ffcf5ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:44686/0 (socket says 172.21.15.32:44686) 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df9044640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf58650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df3fff640 1 -- 172.21.15.32:0/4077129553 learned_addr learned my addr 172.21.15.32:0/4077129553 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 -- 172.21.15.32:0/4077129553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 msgr2=0x564ffcf58040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 -- 172.21.15.32:0/4077129553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 msgr2=0x564ffcf58650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf58650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df8843640 1 -- 172.21.15.32:0/4077129553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ffcf58e50 con 0x564ffcf49620 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.877+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9df8843640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 0x564ffcf5ccd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9de8031a60 tx=0x7f9de8005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d336cfc79a2e8f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:18:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8035040 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9de804f480 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de8040070 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9de80048d0 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 10 client.25538 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564ffcfdbf00 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de8040070 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9de80457e0 con 0x564ffcf49620 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 10 client.25538 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 20 client.25538 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.878+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcfdc500 0x564ffcfde920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.879+0000 7f9dfd7c1740 4 client.25538 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.879+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ffceb6200 con 0x564ffcfdc500 2024-04-23T18:18:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.879+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcfdc500 0x564ffcfde920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.879+0000 7f9df3fff640 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcfdc500 0x564ffcfde920 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aad3fcdbdbae681 server_cookie=4382a7bc5f9afc30 in_seq=0 out_seq=0 2024-04-23T18:18:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.879+0000 7f9dd3fff640 10 client.25538 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dd3fff640 1 -- 172.21.15.32:0/4077129553 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ffceb6200 con 0x564ffcfdc500 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dd3fff640 10 client.25538 handle_command_reply: tid=0 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 client.25538 shutdown 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 msgr2=0x564ffcf5ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 0x564ffcf5ccd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9de8031a60 tx=0x7f9de8005d50 comp rx=0 tx=0).stop 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 shutdown_connections 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564ffcfdc500 0x564ffcfde920 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564ffcf54800 0x564ffcf58650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ffcf49f40 0x564ffcf58040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 --2- 172.21.15.32:0/4077129553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ffcf49620 0x564ffcf5ccd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 >> 172.21.15.32:0/4077129553 conn(0x564ffcf45b30 msgr2=0x564ffcf44cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.892+0000 7f9dd3fff640 0 client.25538 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.893+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 shutdown_connections 2024-04-23T18:18:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.894+0000 7f9dfd7c1740 1 -- 172.21.15.32:0/4077129553 wait complete. 2024-04-23T18:18:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.894+0000 7f9dfd7c1740 20 client.25538 trim_cache size 0 max 16384 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.895+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 msgr2=0x7f9de40a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.895+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 0x7f9de40a6400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dec01bfd0 tx=0x7f9dec01a500 comp rx=0 tx=0).stop 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.895+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 msgr2=0x7f9df419f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.895+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c970 tx=0x7f9de000ce30 comp rx=0 tx=0).stop 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 shutdown_connections 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9df41a3650 0x7f9df41a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9de40a3f40 0x7f9de40a6400 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4107280 0x7f9df419f2d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 --2- 172.21.15.32:0/4259230015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df41021b0 0x7f9df419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.896+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 >> 172.21.15.32:0/4259230015 conn(0x7f9df40fda30 msgr2=0x7f9df40fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.897+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 shutdown_connections 2024-04-23T18:18:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:15.897+0000 7f9dfaaa7640 1 -- 172.21.15.32:0/4259230015 wait complete. 2024-04-23T18:18:16.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:16 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:16.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:16 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:16.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:16 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:17 smithi040 ceph-mon[32626]: pgmap v98: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:18:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:17 smithi064 ceph-mon[33062]: pgmap v98: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:17.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:18:17.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:17 smithi032 ceph-mon[29221]: pgmap v98: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:17.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:17.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:17.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:18:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 170.0 s, 3493.2 tps, lat 28.618 ms stddev 29.271 2024-04-23T18:18:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:19 smithi040 ceph-mon[32626]: pgmap v99: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:19.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:19 smithi064 ceph-mon[33062]: pgmap v99: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:19.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:19 smithi032 ceph-mon[29221]: pgmap v99: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-23T18:18:20.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:20 smithi040 ceph-mon[32626]: pgmap v100: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:18:20.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:20 smithi064 ceph-mon[33062]: pgmap v100: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:18:20.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:20 smithi032 ceph-mon[29221]: pgmap v100: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-23T18:18:22.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:22 smithi032 ceph-mon[29221]: pgmap v101: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 784 op/s 2024-04-23T18:18:22.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:22 smithi040 ceph-mon[32626]: pgmap v101: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 784 op/s 2024-04-23T18:18:22.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:22 smithi064 ceph-mon[33062]: pgmap v101: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 784 op/s 2024-04-23T18:18:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 175.0 s, 3979.0 tps, lat 25.037 ms stddev 27.017 2024-04-23T18:18:24.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:24 smithi032 ceph-mon[29221]: pgmap v102: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-04-23T18:18:24.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:24 smithi040 ceph-mon[32626]: pgmap v102: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-04-23T18:18:24.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:24 smithi064 ceph-mon[33062]: pgmap v102: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-04-23T18:18:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:26 smithi032 ceph-mon[29221]: pgmap v103: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:26.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:26 smithi040 ceph-mon[32626]: pgmap v103: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:26.868 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.1 2024-04-23T18:18:26.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:26.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:26 smithi064 ceph-mon[33062]: pgmap v103: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.041+0000 7ff847e2c640 1 -- 172.21.15.32:0/497190417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108560 msgr2=0x7ff8401089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.041+0000 7ff847e2c640 1 --2- 172.21.15.32:0/497190417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108560 0x7ff8401089e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff830009930 tx=0x7ff83002a440 comp rx=0 tx=0).stop 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.041+0000 7ff847e2c640 1 -- 172.21.15.32:0/497190417 shutdown_connections 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 --2- 172.21.15.32:0/497190417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff840108f20 0x7ff8401153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 --2- 172.21.15.32:0/497190417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108560 0x7ff8401089e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 --2- 172.21.15.32:0/497190417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff84010e620 0x7ff84010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 -- 172.21.15.32:0/497190417 >> 172.21.15.32:0/497190417 conn(0x7ff8400764c0 msgr2=0x7ff8400788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 -- 172.21.15.32:0/497190417 shutdown_connections 2024-04-23T18:18:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.042+0000 7ff847e2c640 1 -- 172.21.15.32:0/497190417 wait complete. 2024-04-23T18:18:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.043+0000 7ff847e2c640 1 Processor -- start 2024-04-23T18:18:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.043+0000 7ff847e2c640 1 -- start start 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff840108560 0x7ff84019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108f20 0x7ff84019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8401188a0 con 0x7ff840108560 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff840118720 con 0x7ff840108f20 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff847e2c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff840118a20 con 0x7ff84010e620 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff8463c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff845bc8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff840108560 0x7ff84019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff8463c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:58296/0 (socket says 172.21.15.32:58296) 2024-04-23T18:18:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.044+0000 7ff8463c9640 1 -- 172.21.15.32:0/1446361815 learned_addr learned my addr 172.21.15.32:0/1446361815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8453c7640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108f20 0x7ff84019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 -- 172.21.15.32:0/1446361815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108f20 msgr2=0x7ff84019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108f20 0x7ff84019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 -- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff840108560 msgr2=0x7ff84019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff840108560 0x7ff84019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 -- 172.21.15.32:0/1446361815 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8401a3d70 con 0x7ff84010e620 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff8463c9640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff83800eb10 tx=0x7ff83800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838004b20 con 0x7ff84010e620 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff838004cc0 con 0x7ff84010e620 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838010670 con 0x7ff84010e620 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8401a4060 con 0x7ff84010e620 2024-04-23T18:18:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.045+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8401ab8f0 con 0x7ff84010e620 2024-04-23T18:18:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.047+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff840068570 con 0x7ff84010e620 2024-04-23T18:18:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.049+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff838005020 con 0x7ff84010e620 2024-04-23T18:18:27.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.049+0000 7ff836ffd640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 0x7ff82c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.049+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff8380d0d40 con 0x7ff84010e620 2024-04-23T18:18:27.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.050+0000 7ff845bc8640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 0x7ff82c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.050+0000 7ff845bc8640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 0x7ff82c0a62a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff83c006fd0 tx=0x7ff83c0090e0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.054+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff83809d3d0 con 0x7ff84010e620 2024-04-23T18:18:27.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.450+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff84002d5c0 con 0x7ff84010e620 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.451+0000 7ff836ffd640 1 -- 172.21.15.32:0/1446361815 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11857 (secure 0 0 0) 0x7ff83809ccd0 con 0x7ff84010e620 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:17:23.797777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:18:27.446 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-23T18:18:27.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 msgr2=0x7ff82c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 0x7ff82c0a62a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff83c006fd0 tx=0x7ff83c0090e0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 msgr2=0x7ff8401a3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff83800eb10 tx=0x7ff83800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 shutdown_connections 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff84010e620 0x7ff8401a3640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff840108f20 0x7ff84019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff82c0a3de0 0x7ff82c0a62a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 --2- 172.21.15.32:0/1446361815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff840108560 0x7ff84019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 >> 172.21.15.32:0/1446361815 conn(0x7ff8400764c0 msgr2=0x7ff840077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 shutdown_connections 2024-04-23T18:18:27.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.456+0000 7ff847e2c640 1 -- 172.21.15.32:0/1446361815 wait complete. 2024-04-23T18:18:27.467 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 94.1 --await-for 42.0' 2024-04-23T18:18:27.467 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 94.1 --await-for 42.0 2024-04-23T18:18:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.660+0000 7fdf867fc640 1 -- 172.21.15.32:0/3742605597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c035020 con 0x7fdf881091c0 2024-04-23T18:18:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.660+0000 7fdf8de72640 1 -- 172.21.15.32:0/3742605597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf881091c0 msgr2=0x7fdf881095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.660+0000 7fdf8de72640 1 --2- 172.21.15.32:0/3742605597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf881091c0 0x7fdf881095a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009900 tx=0x7fdf7c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.661+0000 7fdf8de72640 1 -- 172.21.15.32:0/3742605597 shutdown_connections 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.661+0000 7fdf8de72640 1 --2- 172.21.15.32:0/3742605597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf8811b470 0x7fdf8811d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.661+0000 7fdf8de72640 1 --2- 172.21.15.32:0/3742605597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf88109ae0 0x7fdf8811aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.661+0000 7fdf8de72640 1 --2- 172.21.15.32:0/3742605597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf881091c0 0x7fdf881095a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.661+0000 7fdf8de72640 1 -- 172.21.15.32:0/3742605597 >> 172.21.15.32:0/3742605597 conn(0x7fdf8806bf40 msgr2=0x7fdf8806e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.662+0000 7fdf8de72640 1 -- 172.21.15.32:0/3742605597 shutdown_connections 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.662+0000 7fdf8de72640 1 -- 172.21.15.32:0/3742605597 wait complete. 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 Processor -- start 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 -- start start 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf881091c0 0x7fdf88118770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf88109ae0 0x7fdf88110730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdf881210d0 con 0x7fdf8811b470 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf88120f50 con 0x7fdf881091c0 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.663+0000 7fdf8de72640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdf88121250 con 0x7fdf88109ae0 2024-04-23T18:18:27.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49464/0 (socket says 172.21.15.32:49464) 2024-04-23T18:18:27.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 -- 172.21.15.32:0/1365885183 learned_addr learned my addr 172.21.15.32:0/1365885183 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 -- 172.21.15.32:0/1365885183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf88109ae0 msgr2=0x7fdf88110730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf88109ae0 0x7fdf88110730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 -- 172.21.15.32:0/1365885183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf881091c0 msgr2=0x7fdf88118770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf881091c0 0x7fdf88118770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.664+0000 7fdf87fff640 1 -- 172.21.15.32:0/1365885183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf88111470 con 0x7fdf8811b470 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf87fff640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009fd0 tx=0x7fdf7800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf84ff9640 1 -- 172.21.15.32:0/1365885183 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78011830 con 0x7fdf8811b470 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf88079020 con 0x7fdf8811b470 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf88079570 con 0x7fdf8811b470 2024-04-23T18:18:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf84ff9640 1 -- 172.21.15.32:0/1365885183 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf78016070 con 0x7fdf8811b470 2024-04-23T18:18:27.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.665+0000 7fdf84ff9640 1 -- 172.21.15.32:0/1365885183 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78011830 con 0x7fdf8811b470 2024-04-23T18:18:27.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.669+0000 7fdf84ff9640 1 -- 172.21.15.32:0/1365885183 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fdf78010cf0 con 0x7fdf8811b470 2024-04-23T18:18:27.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.670+0000 7fdf84ff9640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 0x7fdf700a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.671+0000 7fdf84ff9640 1 -- 172.21.15.32:0/1365885183 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fdf780cf990 con 0x7fdf8811b470 2024-04-23T18:18:27.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.671+0000 7fdf877fe640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 0x7fdf700a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.672+0000 7fdf877fe640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 0x7fdf700a64e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0096c0 tx=0x7fdf7c005d50 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.673+0000 7fdf90b84740 1 Processor -- start 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.673+0000 7fdf90b84740 1 -- start start 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf2e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf38b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c9be93370 con 0x558c9bf39090 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558c9be302e0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf90b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c9be8f330 con 0x558c9bf2e0a0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf38b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf38b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49802/0 (socket says 172.21.15.32:49802) 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf86ffd640 1 -- 172.21.15.32:0/4172359803 learned_addr learned my addr 172.21.15.32:0/4172359803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.674+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf2e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 -- 172.21.15.32:0/4172359803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 msgr2=0x558c9bf2e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf2e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 -- 172.21.15.32:0/4172359803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 msgr2=0x558c9bf38b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf38b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 -- 172.21.15.32:0/4172359803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9bc71d80 con 0x558c9bf39090 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3b480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fdf780cd1e0 tx=0x7fdf7800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8583c711099a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf6dffb640 1 -- 172.21.15.32:0/4172359803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7809ba30 con 0x558c9bf39090 2024-04-23T18:18:27.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf6dffb640 1 -- 172.21.15.32:0/4172359803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7809fde0 con 0x558c9bf39090 2024-04-23T18:18:27.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.675+0000 7fdf6dffb640 1 -- 172.21.15.32:0/4172359803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780cd350 con 0x558c9bf39090 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.676+0000 7fdf90b84740 1 -- 172.21.15.32:0/4172359803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 msgr2=0x558c9bf3b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.676+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3b480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fdf780cd1e0 tx=0x7fdf7800bf90 comp rx=0 tx=0).stop 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.677+0000 7fdf90b84740 1 -- 172.21.15.32:0/4172359803 shutdown_connections 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.677+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3b480 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.677+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf38b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.677+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4172359803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf2e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.677+0000 7fdf90b84740 1 -- 172.21.15.32:0/4172359803 >> 172.21.15.32:0/4172359803 conn(0x558c9bf2a3c0 msgr2=0x558c9bf2a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.679+0000 7fdf90b84740 1 -- 172.21.15.32:0/4172359803 shutdown_connections 2024-04-23T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.679+0000 7fdf90b84740 1 -- 172.21.15.32:0/4172359803 wait complete. 2024-04-23T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 -- start start 2024-04-23T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 Processor -- start 2024-04-23T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf36560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf36aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c9bf3be20 con 0x558c9bf39090 2024-04-23T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558c9be2dfb0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c9bf3bfa0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf90b84740 10 client.0 _ll_register_callbacks cb 0x558c9be2d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.680+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf36560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:58338/0 (socket says 172.21.15.32:58338) 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf877fe640 1 -- 172.21.15.32:0/4023209661 learned_addr learned my addr 172.21.15.32:0/4023209661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf36aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 -- 172.21.15.32:0/4023209661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 msgr2=0x558c9bf36020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 -- 172.21.15.32:0/4023209661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 msgr2=0x558c9bf36aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf36aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 -- 172.21.15.32:0/4023209661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9bcd2510 con 0x558c9bf2e9c0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf36560 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b810 tx=0x7fdf8000bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f9024032219fe9b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000add0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fdf80014420 con 0x558c9bf2e9c0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.681+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7809fde0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdf80014620 con 0x558c9bf2e9c0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 10 client.25550 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c9bb65670 con 0x558c9bf2e9c0 2024-04-23T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7809fde0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdf8001d7c0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 10 client.25550 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 10 client.25550 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 20 client.25550 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bf49540 0x558c9bf4b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 4 client.25550 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.682+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c9bea0d90 con 0x558c9bf49540 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.684+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bf49540 0x558c9bf4b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.684+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bf49540 0x558c9bf4b950 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65c0a90f0038917 server_cookie=1e48b5d1f60f656b in_seq=0 out_seq=0 2024-04-23T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.684+0000 7fdf66ffd640 10 client.25550 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:27.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf66ffd640 1 -- 172.21.15.32:0/4023209661 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c9bea0d90 con 0x558c9bf49540 2024-04-23T18:18:27.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf66ffd640 10 client.25550 handle_command_reply: tid=0 2024-04-23T18:18:27.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 client.25550 shutdown 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 msgr2=0x558c9bf36560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf36560 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b810 tx=0x7fdf8000bcd0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 shutdown_connections 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bf49540 0x558c9bf4b950 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf36aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e9c0 0x558c9bf36560 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf66ffd640 0 client.25550 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4023209661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e0a0 0x558c9bf36020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 >> 172.21.15.32:0/4023209661 conn(0x558c9bf2a3c0 msgr2=0x558c9bf29600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 shutdown_connections 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 1 -- 172.21.15.32:0/4023209661 wait complete. 2024-04-23T18:18:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.685+0000 7fdf90b84740 20 client.25550 trim_cache size 0 max 16384 2024-04-23T18:18:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.711+0000 7fdf90b84740 1 Processor -- start 2024-04-23T18:18:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.711+0000 7fdf90b84740 1 -- start start 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 0x558c9bf30840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf39090 0x558c9bf51260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49822/0 (socket says 172.21.15.32:49822) 2024-04-23T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf86ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 0x558c9bf30840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf877fe640 1 -- 172.21.15.32:0/4013299370 learned_addr learned my addr 172.21.15.32:0/4013299370 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdf74010fc0 con 0x558c9bf2e9c0 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7400e7d0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.712+0000 7fdf87fff640 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf39090 0x558c9bf51260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7400c620 con 0x558c9bf39090 2024-04-23T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 -- 172.21.15.32:0/4013299370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf39090 msgr2=0x558c9bf51260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf39090 0x558c9bf51260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 -- 172.21.15.32:0/4013299370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 msgr2=0x558c9bf30840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 0x558c9bf30840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 -- 172.21.15.32:0/4013299370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9bf34fd0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 0x558c9bf30840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf877fe640 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c005ab0 tx=0x7fdf7c03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252a521fe6428506 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf6d7fa640 1 -- 172.21.15.32:0/4013299370 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c035040 con 0x558c9bf2e0a0 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf6d7fa640 1 -- 172.21.15.32:0/4013299370 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7c040040 con 0x558c9bf2e0a0 2024-04-23T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.713+0000 7fdf6d7fa640 1 -- 172.21.15.32:0/4013299370 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c051890 con 0x558c9bf2e0a0 2024-04-23T18:18:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.714+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 msgr2=0x558c9bf44d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.714+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c005ab0 tx=0x7fdf7c03e000 comp rx=0 tx=0).stop 2024-04-23T18:18:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.716+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 shutdown_connections 2024-04-23T18:18:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.716+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf39090 0x558c9bf51260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.716+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf2e9c0 0x558c9bf30840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.716+0000 7fdf90b84740 1 --2- 172.21.15.32:0/4013299370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf44d40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.716+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 >> 172.21.15.32:0/4013299370 conn(0x558c9bf2a3c0 msgr2=0x558c9bf37850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.717+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 shutdown_connections 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.717+0000 7fdf90b84740 1 -- 172.21.15.32:0/4013299370 wait complete. 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.717+0000 7fdf90b84740 1 -- start start 2024-04-23T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.717+0000 7fdf90b84740 1 Processor -- start 2024-04-23T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.718+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 0x558c9bf41760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7400cc90 con 0x558c9bf39090 2024-04-23T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7400e100 con 0x558c9bf2e0a0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7400d360 con 0x558c9bf2e9c0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf90b84740 10 client.0 _ll_register_callbacks cb 0x7fdf8000b450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf86ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 0x558c9bf41760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49832/0 (socket says 172.21.15.32:49832) 2024-04-23T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 -- 172.21.15.32:0/2721200885 learned_addr learned my addr 172.21.15.32:0/2721200885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 -- 172.21.15.32:0/2721200885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 msgr2=0x558c9bf41760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 0x558c9bf41760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 -- 172.21.15.32:0/2721200885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 msgr2=0x558c9bf3d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 -- 172.21.15.32:0/2721200885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9bf3dfa0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 0x558c9bf41760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf877fe640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0044a0 tx=0x7fdf7c032a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e615da91b27f5e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.719+0000 7fdf66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c051590 con 0x558c9bf2e0a0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fdf7c051730 con 0x558c9bf2e0a0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7c040040 con 0x558c9bf2e0a0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdf7c067050 con 0x558c9bf2e0a0 2024-04-23T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 10 client.25556 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c9bfb06d0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7c05e070 con 0x558c9bf2e0a0 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdf7c003bc0 con 0x558c9bf2e0a0 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 10 client.25556 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 10 client.25556 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 20 client.25556 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bfb0ec0 0x558c9bfb32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 4 client.25556 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 94.1, "await_for": 42.0}] 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.720+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 94.1, "await_for": 42.0}) -- 0x7fdf7c040040 con 0x558c9bfb0ec0 2024-04-23T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.722+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bfb0ec0 0x558c9bfb32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.724+0000 7fdf66ffd640 10 client.25556 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:27.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.724+0000 7fdf86ffd640 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bfb0ec0 0x558c9bfb32e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18a3e4eefdae114 server_cookie=86270aa141bbcab0 in_seq=0 out_seq=0 2024-04-23T18:18:27.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:27 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1446361815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:27.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:27 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1446361815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:27.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:27 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1446361815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf66ffd640 1 -- 172.21.15.32:0/2721200885 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7fdf7c040040 con 0x558c9bfb0ec0 2024-04-23T18:18:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf66ffd640 10 client.25556 handle_command_reply: tid=0 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 client.25556 shutdown 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 msgr2=0x558c9bf41220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0044a0 tx=0x7fdf7c032a10 comp rx=0 tx=0).stop 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 shutdown_connections 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c9bfb0ec0 0x558c9bfb32e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c9bf39090 0x558c9bf3d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c9bf2e9c0 0x558c9bf41760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf66ffd640 0 client.25556 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 --2- 172.21.15.32:0/2721200885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c9bf2e0a0 0x558c9bf41220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 >> 172.21.15.32:0/2721200885 conn(0x558c9bf2a3c0 msgr2=0x558c9bf380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 shutdown_connections 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 1 -- 172.21.15.32:0/2721200885 wait complete. 2024-04-23T18:18:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.939+0000 7fdf90b84740 20 client.25556 trim_cache size 0 max 16384 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 28, 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 10, 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "d7c1a45b": { 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 10, 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:18:27.935 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 94.1, 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:18:27.936 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:18:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 msgr2=0x7fdf700a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 0x7fdf700a64e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0096c0 tx=0x7fdf7c005d50 comp rx=0 tx=0).stop 2024-04-23T18:18:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 msgr2=0x7fdf88110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009fd0 tx=0x7fdf7800b8e0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 shutdown_connections 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fdf700a4020 0x7fdf700a64e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8811b470 0x7fdf88110c70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf88109ae0 0x7fdf88110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 --2- 172.21.15.32:0/1365885183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf881091c0 0x7fdf88118770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 >> 172.21.15.32:0/1365885183 conn(0x7fdf8806bf40 msgr2=0x7fdf88119dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 shutdown_connections 2024-04-23T18:18:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:18:27.940+0000 7fdf8de72640 1 -- 172.21.15.32:0/1365885183 wait complete. 2024-04-23T18:18:27.945 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd7c1a45b', quiesce took 1.1 seconds. Will release after: 34.1 2024-04-23T18:18:28.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:28 smithi032 ceph-mon[29221]: pgmap v104: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 180.0 s, 2535.8 tps, lat 30.681 ms stddev 30.979 2024-04-23T18:18:28.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:28 smithi040 ceph-mon[32626]: pgmap v104: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:28.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:28 smithi064 ceph-mon[33062]: pgmap v104: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 -- 172.21.15.32:0/1993563123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8840742f0 msgr2=0x7fa8840746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 --2- 172.21.15.32:0/1993563123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8840742f0 0x7fa8840746d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa880009900 tx=0x7fa88002a480 comp rx=0 tx=0).stop 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 -- 172.21.15.32:0/1993563123 shutdown_connections 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 --2- 172.21.15.32:0/1993563123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa884112f60 0x7fa884115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 --2- 172.21.15.32:0/1993563123 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884074c10 0x7fa884112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 --2- 172.21.15.32:0/1993563123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8840742f0 0x7fa8840746d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 -- 172.21.15.32:0/1993563123 >> 172.21.15.32:0/1993563123 conn(0x7fa8840fda40 msgr2=0x7fa8840ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 -- 172.21.15.32:0/1993563123 shutdown_connections 2024-04-23T18:18:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.001+0000 7fa88b631640 1 -- 172.21.15.32:0/1993563123 wait complete. 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 Processor -- start 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 -- start start 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840742f0 0x7fa88419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884112f60 0x7fa8841a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841187e0 con 0x7fa884074c10 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa884118660 con 0x7fa8840742f0 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa88b631640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa884118960 con 0x7fa884112f60 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa888bcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa888bcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49544/0 (socket says 172.21.15.32:49544) 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.002+0000 7fa888bcc640 1 -- 172.21.15.32:0/771678947 learned_addr learned my addr 172.21.15.32:0/771678947 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa889bce640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884112f60 0x7fa8841a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa8893cd640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840742f0 0x7fa88419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 -- 172.21.15.32:0/771678947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884112f60 msgr2=0x7fa8841a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884112f60 0x7fa8841a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 -- 172.21.15.32:0/771678947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840742f0 msgr2=0x7fa88419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840742f0 0x7fa88419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 -- 172.21.15.32:0/771678947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8841a3d10 con 0x7fa884074c10 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa888bcc640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c920 tx=0x7fa87400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa87a7fc640 1 -- 172.21.15.32:0/771678947 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874007c70 con 0x7fa884074c10 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa87a7fc640 1 -- 172.21.15.32:0/771678947 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa874007e10 con 0x7fa884074c10 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8841a4000 con 0x7fa884074c10 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa87a7fc640 1 -- 172.21.15.32:0/771678947 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874018820 con 0x7fa884074c10 2024-04-23T18:18:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.003+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8841ab890 con 0x7fa884074c10 2024-04-23T18:18:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.005+0000 7fa87a7fc640 1 -- 172.21.15.32:0/771678947 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa87400f450 con 0x7fa884074c10 2024-04-23T18:18:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 Processor -- start 2024-04-23T18:18:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 -- start start 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa87a7fc640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 0x7fa8700a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc93562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560bc92b0970 con 0x560bc9356800 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560bc924d4a0 con 0x560bc934bf40 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.006+0000 7fa88e429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560bc92ac930 con 0x560bc934b620 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa889bce640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 0x7fa8700a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa87a7fc640 1 -- 172.21.15.32:0/771678947 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa8740d08e0 con 0x7fa884074c10 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa889bce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:58380/0 (socket says 172.21.15.32:58380) 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa889bce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49560/0 (socket says 172.21.15.32:49560) 2024-04-23T18:18:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 -- 172.21.15.32:0/3089215588 learned_addr learned my addr 172.21.15.32:0/3089215588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa888bcc640 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc93562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 -- 172.21.15.32:0/3089215588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 msgr2=0x560bc93562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc93562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 -- 172.21.15.32:0/3089215588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 msgr2=0x560bc9358bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 -- 172.21.15.32:0/3089215588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc8fc3210 con 0x560bc934b620 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa889bce640 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa889bce640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 0x7fa8700a61d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009fd0 tx=0x7fa87c009da0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.007+0000 7fa8893cd640 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa88002a9a0 tx=0x7fa880032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797046080e4c1b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa86f7fe640 1 -- 172.21.15.32:0/3089215588 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88003f070 con 0x560bc934b620 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa86f7fe640 1 -- 172.21.15.32:0/3089215588 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa880032e50 con 0x560bc934b620 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa86f7fe640 1 -- 172.21.15.32:0/3089215588 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880035070 con 0x560bc934b620 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 -- 172.21.15.32:0/3089215588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 msgr2=0x560bc934ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa88002a9a0 tx=0x7fa880032ce0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 -- 172.21.15.32:0/3089215588 shutdown_connections 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc9358bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc93562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 --2- 172.21.15.32:0/3089215588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934b620 0x560bc934ba00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 -- 172.21.15.32:0/3089215588 >> 172.21.15.32:0/3089215588 conn(0x560bc9347b30 msgr2=0x560bc9347f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.008+0000 7fa88e429740 1 -- 172.21.15.32:0/3089215588 shutdown_connections 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 -- 172.21.15.32:0/3089215588 wait complete. 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 -- start start 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 Processor -- start 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9353440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc93539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560bc93593d0 con 0x560bc934b620 2024-04-23T18:18:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560bc924b0b0 con 0x560bc934bf40 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560bc9359550 con 0x560bc9356800 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa88e429740 10 client.0 _ll_register_callbacks cb 0x560bc9238d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.009+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49566/0 (socket says 172.21.15.32:49566) 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 -- 172.21.15.32:0/931670872 learned_addr learned my addr 172.21.15.32:0/931670872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa889bce640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc93539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa888bcc640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9353440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 -- 172.21.15.32:0/931670872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 msgr2=0x560bc93539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc93539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 -- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 msgr2=0x560bc9353440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9353440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 -- 172.21.15.32:0/931670872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc908f620 con 0x560bc934b620 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa888bcc640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9353440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa8893cd640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa880006010 tx=0x7fa880031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a031b955c03904b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88004b060 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa880032e50 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa880004440 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa88003f030 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa88e429740 10 client.15921 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.010+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560bc9365300 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa880032e50 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa8800049f0 con 0x560bc934b620 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 10 client.15921 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 20 client.15921 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc9366650 0x560bc9368a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 4 client.15921 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bc92be390 con 0x560bc9366650 2024-04-23T18:18:29.011 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.011+0000 7fa888bcc640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc9366650 0x560bc9368a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.011 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa86cff9640 10 client.15921 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:29.011 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa888bcc640 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc9366650 0x560bc9368a40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf1ce027a5436ef server_cookie=a333d460b2ab79fd in_seq=0 out_seq=0 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa86cff9640 1 -- 172.21.15.32:0/931670872 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560bc92be390 con 0x560bc9366650 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa86cff9640 10 client.15921 handle_command_reply: tid=0 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa88e429740 1 client.15921 shutdown 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.012+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 msgr2=0x560bc9356150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa880006010 tx=0x7fa880031750 comp rx=0 tx=0).stop 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 shutdown_connections 2024-04-23T18:18:29.012 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc9366650 0x560bc9368a40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc93539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9353440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa86cff9640 0 client.15921 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 --2- 172.21.15.32:0/931670872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9356150 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 >> 172.21.15.32:0/931670872 conn(0x560bc9347b30 msgr2=0x560bc934d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 shutdown_connections 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 1 -- 172.21.15.32:0/931670872 wait complete. 2024-04-23T18:18:29.013 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.013+0000 7fa88e429740 20 client.15921 trim_cache size 0 max 16384 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 Processor -- start 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 -- start start 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9355c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc936e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b69f0 con 0x560bc934b620 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b4200 con 0x560bc934bf40 2024-04-23T18:18:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa88e429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b2050 con 0x560bc9356800 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa8893cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49570/0 (socket says 172.21.15.32:49570) 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa8893cd640 1 -- 172.21.15.32:0/1897636024 learned_addr learned my addr 172.21.15.32:0/1897636024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa888bcc640 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9355c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.028+0000 7fa889bce640 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc936e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 -- 172.21.15.32:0/1897636024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 msgr2=0x560bc936e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc936e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 -- 172.21.15.32:0/1897636024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 msgr2=0x560bc9355c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9355c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 -- 172.21.15.32:0/1897636024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc93477f0 con 0x560bc934b620 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa8800052b0 tx=0x7fa88002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c582a3f0f1ab11c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa86effd640 1 -- 172.21.15.32:0/1897636024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88004b040 con 0x560bc934b620 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa86effd640 1 -- 172.21.15.32:0/1897636024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa880004b70 con 0x560bc934b620 2024-04-23T18:18:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.029+0000 7fa86effd640 1 -- 172.21.15.32:0/1897636024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880035040 con 0x560bc934b620 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- 172.21.15.32:0/1897636024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 msgr2=0x560bc9362290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa8800052b0 tx=0x7fa88002a480 comp rx=0 tx=0).stop 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- 172.21.15.32:0/1897636024 shutdown_connections 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc9356800 0x560bc936e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934bf40 0x560bc9355c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- 172.21.15.32:0/1897636024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc934b620 0x560bc9362290 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- 172.21.15.32:0/1897636024 >> 172.21.15.32:0/1897636024 conn(0x560bc9347b30 msgr2=0x560bc934e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- 172.21.15.32:0/1897636024 shutdown_connections 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- 172.21.15.32:0/1897636024 wait complete. 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- start start 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 Processor -- start 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934b620 0x560bc935a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934bf40 0x560bc935a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b26c0 con 0x560bc9356800 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b3b30 con 0x560bc934b620 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.030+0000 7fa88e429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841b2d90 con 0x560bc934bf40 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa88e429740 10 client.0 _ll_register_callbacks cb 0x7fa880032bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49578/0 (socket says 172.21.15.32:49578) 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 -- 172.21.15.32:0/1599762137 learned_addr learned my addr 172.21.15.32:0/1599762137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa888bcc640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934bf40 0x560bc935a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934b620 0x560bc935a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 -- 172.21.15.32:0/1599762137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934bf40 msgr2=0x560bc935a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934bf40 0x560bc935a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 -- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934b620 msgr2=0x560bc935a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934b620 0x560bc935a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 -- 172.21.15.32:0/1599762137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc93cb880 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa889bce640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0098b0 tx=0x7fa87c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292e29ea06cb892e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c003e00 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa87c01b040 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa880004b70 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa87c004710 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa88e429740 10 client.15927 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.031+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560bc93ccb60 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa880004b70 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa87c01f070 con 0x560bc9356800 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 10 client.15927 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 20 client.15927 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc93cd380 0x560bc93cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 4 client.15927 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bc92b8200 con 0x560bc93cd380 2024-04-23T18:18:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.032+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc93cd380 0x560bc93cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.033+0000 7fa8893cd640 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc93cd380 0x560bc93cf7a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b2bbc159c00e7c server_cookie=28cb1ef7bb90472f in_seq=0 out_seq=0 2024-04-23T18:18:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.033+0000 7fa86cff9640 10 client.15927 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa86cff9640 1 -- 172.21.15.32:0/1599762137 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bc92b8200 con 0x560bc93cd380 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa86cff9640 10 client.15927 handle_command_reply: tid=0 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 client.15927 shutdown 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 msgr2=0x560bc935acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0098b0 tx=0x7fa87c009600 comp rx=0 tx=0).stop 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 shutdown_connections 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560bc93cd380 0x560bc93cf7a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560bc9356800 0x560bc935acb0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560bc934bf40 0x560bc935a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 --2- 172.21.15.32:0/1599762137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bc934b620 0x560bc935a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa86cff9640 0 client.15927 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.048+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 >> 172.21.15.32:0/1599762137 conn(0x560bc9347b30 msgr2=0x560bc9361280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 shutdown_connections 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88e429740 1 -- 172.21.15.32:0/1599762137 wait complete. 2024-04-23T18:18:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88e429740 20 client.15927 trim_cache size 0 max 16384 2024-04-23T18:18:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:18:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:18:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:18:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 msgr2=0x7fa8700a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 0x7fa8700a61d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009fd0 tx=0x7fa87c009da0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 msgr2=0x7fa88419f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c920 tx=0x7fa87400cde0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 shutdown_connections 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa884112f60 0x7fa8841a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa8700a3d10 0x7fa8700a61d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa884074c10 0x7fa88419f260 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 --2- 172.21.15.32:0/771678947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840742f0 0x7fa88419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 >> 172.21.15.32:0/771678947 conn(0x7fa8840fda40 msgr2=0x7fa8840ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 shutdown_connections 2024-04-23T18:18:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:29.049+0000 7fa88b631640 1 -- 172.21.15.32:0/771678947 wait complete. 2024-04-23T18:18:30.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:30 smithi032 ceph-mon[29221]: pgmap v105: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 10 MiB/s wr, 925 op/s 2024-04-23T18:18:30.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:30 smithi040 ceph-mon[32626]: pgmap v105: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 10 MiB/s wr, 925 op/s 2024-04-23T18:18:30.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:30 smithi064 ceph-mon[33062]: pgmap v105: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 10 MiB/s wr, 925 op/s 2024-04-23T18:18:32.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:32 smithi032 ceph-mon[29221]: pgmap v106: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 602 op/s 2024-04-23T18:18:32.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:32 smithi040 ceph-mon[32626]: pgmap v106: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 602 op/s 2024-04-23T18:18:32.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:32 smithi064 ceph-mon[33062]: pgmap v106: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 602 op/s 2024-04-23T18:18:33.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 185.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:34.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:34 smithi032 ceph-mon[29221]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-04-23T18:18:34.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:34 smithi040 ceph-mon[32626]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-04-23T18:18:34.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:34 smithi064 ceph-mon[33062]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-04-23T18:18:36.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:36 smithi032 ceph-mon[29221]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:36.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:36 smithi040 ceph-mon[32626]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:36.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:36 smithi064 ceph-mon[33062]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:38.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:38 smithi032 ceph-mon[29221]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 190.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:38.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:38 smithi040 ceph-mon[32626]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:38.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:38 smithi064 ceph-mon[33062]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-04-23T18:18:40.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:40 smithi040 ceph-mon[32626]: pgmap v110: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 744 op/s 2024-04-23T18:18:40.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:40 smithi064 ceph-mon[33062]: pgmap v110: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 744 op/s 2024-04-23T18:18:41.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:40 smithi032 ceph-mon[29221]: pgmap v110: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 744 op/s 2024-04-23T18:18:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/4073030445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 msgr2=0x7fa3c8115350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/4073030445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c8115350 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009ca0 tx=0x7fa3c002a950 comp rx=0 tx=0).stop 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/4073030445 shutdown_connections 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/4073030445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c8115350 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/4073030445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c8112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/4073030445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 0x7fa3c80746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/4073030445 >> 172.21.15.32:0/4073030445 conn(0x7fa3c80fda40 msgr2=0x7fa3c80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/4073030445 shutdown_connections 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.148+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/4073030445 wait complete. 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 Processor -- start 2024-04-23T18:18:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 -- start start 2024-04-23T18:18:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 0x7fa3c819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81187b0 con 0x7fa3c8112f60 2024-04-23T18:18:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8118630 con 0x7fa3c8074c10 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3cfc2a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8118930 con 0x7fa3c80742f0 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.149+0000 7fa3ce1c7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35518/0 (socket says 172.21.15.32:35518) 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 -- 172.21.15.32:0/3430501764 learned_addr learned my addr 172.21.15.32:0/3430501764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 0x7fa3c819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 -- 172.21.15.32:0/3430501764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 msgr2=0x7fa3c819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 0x7fa3c819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 -- 172.21.15.32:0/3430501764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 msgr2=0x7fa3c819f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c819f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 -- 172.21.15.32:0/3430501764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c81a3da0 con 0x7fa3c8112f60 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3ce1c7640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa3c002ae70 tx=0x7fa3c002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3beffd640 1 -- 172.21.15.32:0/3430501764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0034040 con 0x7fa3c8112f60 2024-04-23T18:18:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3beffd640 1 -- 172.21.15.32:0/3430501764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c00308f0 con 0x7fa3c8112f60 2024-04-23T18:18:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c81a4030 con 0x7fa3c8112f60 2024-04-23T18:18:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3beffd640 1 -- 172.21.15.32:0/3430501764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00047c0 con 0x7fa3c8112f60 2024-04-23T18:18:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.150+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c81ab920 con 0x7fa3c8112f60 2024-04-23T18:18:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.153+0000 7fa3beffd640 1 -- 172.21.15.32:0/3430501764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa3c0037020 con 0x7fa3c8112f60 2024-04-23T18:18:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.153+0000 7fa3d2a1a740 1 Processor -- start 2024-04-23T18:18:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.153+0000 7fa3d2a1a740 1 -- start start 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce6f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6ff800 0x55fdce701bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fdce659970 con 0x55fdce6f4f40 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fdce5f64a0 con 0x55fdce6f4620 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fdce655930 con 0x55fdce6ff800 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3beffd640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 0x7fa3b40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35520/0 (socket says 172.21.15.32:35520) 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/2824707007 learned_addr learned my addr 172.21.15.32:0/2824707007 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3ce1c7640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 0x7fa3b40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6ff800 0x55fdce701bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce6f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3beffd640 1 -- 172.21.15.32:0/3430501764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa3c00f1620 con 0x7fa3c8112f60 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/2824707007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6ff800 msgr2=0x55fdce701bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6ff800 0x55fdce701bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/2824707007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 msgr2=0x55fdce6f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce6f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/2824707007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdce36c210 con 0x55fdce6f4f40 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3ce1c7640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 0x7fa3b40a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00eef80 tx=0x7fa3c00095c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.154+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400bd10 tx=0x7fa3c40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16af3664377b7f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3b3fff640 1 -- 172.21.15.32:0/2824707007 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4004030 con 0x55fdce6f4f40 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3b3fff640 1 -- 172.21.15.32:0/2824707007 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c40035f0 con 0x55fdce6f4f40 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3b3fff640 1 -- 172.21.15.32:0/2824707007 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40117b0 con 0x55fdce6f4f40 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/2824707007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 msgr2=0x55fdce6ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400bd10 tx=0x7fa3c40107a0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/2824707007 shutdown_connections 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6ff800 0x55fdce701bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce6ff2c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/2824707007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce6f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/2824707007 >> 172.21.15.32:0/2824707007 conn(0x55fdce6f0b30 msgr2=0x55fdce6f0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.155+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/2824707007 shutdown_connections 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/2824707007 wait complete. 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 -- start start 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 Processor -- start 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce6ff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4f40 0x55fdce6fc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fdce702430 con 0x55fdce6ff800 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fdce5f40b0 con 0x55fdce6f4f40 2024-04-23T18:18:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fdce7025b0 con 0x55fdce6f4620 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.156+0000 7fa3d2a1a740 10 client.0 _ll_register_callbacks cb 0x55fdce5e1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35534/0 (socket says 172.21.15.32:35534) 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/342406864 learned_addr learned my addr 172.21.15.32:0/342406864 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4f40 0x55fdce6fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce6ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/342406864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 msgr2=0x55fdce6ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce6ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4f40 msgr2=0x55fdce6fc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4f40 0x55fdce6fc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/342406864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdce438620 con 0x55fdce6ff800 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800f440 tx=0x7fa3b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9452c55f3b5b88 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b801c040 con 0x55fdce6ff800 2024-04-23T18:18:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa3b8013220 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.157+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c40035f0 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3b8029050 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 10 client.15939 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fdce70e400 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c40035f0 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa3b8003990 con 0x55fdce6ff800 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 10 client.15939 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 20 client.15939 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce70f8c0 0x55fdce711cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 4 client.15939 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdce667390 con 0x55fdce70f8c0 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.158+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce70f8c0 0x55fdce711cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.159+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce70f8c0 0x55fdce711cb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=448a0bc7f3f00cb7 server_cookie=bf729755fa93926d in_seq=0 out_seq=0 2024-04-23T18:18:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.159+0000 7fa3b17fa640 10 client.15939 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.159+0000 7fa3b17fa640 1 -- 172.21.15.32:0/342406864 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdce667390 con 0x55fdce70f8c0 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.159+0000 7fa3b17fa640 10 client.15939 handle_command_reply: tid=0 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.159+0000 7fa3d2a1a740 1 client.15939 shutdown 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 msgr2=0x55fdce6fc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800f440 tx=0x7fa3b80046c0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 shutdown_connections 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce70f8c0 0x55fdce711cb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce6fc9c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4f40 0x55fdce6fc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/342406864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce6ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3b17fa640 0 client.15939 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 >> 172.21.15.32:0/342406864 conn(0x55fdce6f0b30 msgr2=0x55fdce6f68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 shutdown_connections 2024-04-23T18:18:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/342406864 wait complete. 2024-04-23T18:18:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.160+0000 7fa3d2a1a740 20 client.15939 trim_cache size 0 max 16384 2024-04-23T18:18:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 Processor -- start 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 -- start start 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce70b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 0x55fdce6efb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b6a40 con 0x55fdce6ff800 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b4250 con 0x55fdce6f4620 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b20a0 con 0x55fdce6f4f40 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3cd1c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3cd1c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35538/0 (socket says 172.21.15.32:35538) 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/83549152 learned_addr learned my addr 172.21.15.32:0/83549152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 0x55fdce6efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.175+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce70b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/83549152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 msgr2=0x55fdce6efb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 0x55fdce6efb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/83549152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 msgr2=0x55fdce70b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce70b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 -- 172.21.15.32:0/83549152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdce6f4d70 con 0x55fdce6ff800 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 0x55fdce6efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8027040 tx=0x7fa3b8005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390b32e13846d77f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3b37fe640 1 -- 172.21.15.32:0/83549152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8032020 con 0x55fdce6ff800 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3b37fe640 1 -- 172.21.15.32:0/83549152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3b8013550 con 0x55fdce6ff800 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.176+0000 7fa3b37fe640 1 -- 172.21.15.32:0/83549152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8022440 con 0x55fdce6ff800 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/83549152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 msgr2=0x55fdce7175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8027040 tx=0x7fa3b8005970 comp rx=0 tx=0).stop 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/83549152 shutdown_connections 2024-04-23T18:18:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6ff800 0x55fdce7175c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4f40 0x55fdce6efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/83549152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6f4620 0x55fdce70b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/83549152 >> 172.21.15.32:0/83549152 conn(0x55fdce6f0b30 msgr2=0x55fdce6ef420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/83549152 shutdown_connections 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/83549152 wait complete. 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 -- start start 2024-04-23T18:18:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.177+0000 7fa3d2a1a740 1 Processor -- start 2024-04-23T18:18:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce707be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6ff800 0x55fdce7054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b2710 con 0x55fdce6f4f40 2024-04-23T18:18:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b3b80 con 0x55fdce6ff800 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81b2de0 con 0x55fdce6f4620 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 10 client.0 _ll_register_callbacks cb 0x7fa3b8013c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35550/0 (socket says 172.21.15.32:35550) 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/248321925 learned_addr learned my addr 172.21.15.32:0/248321925 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce707be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/248321925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 msgr2=0x55fdce707be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce707be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6ff800 msgr2=0x55fdce7054b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6ff800 0x55fdce7054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 -- 172.21.15.32:0/248321925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdce705cb0 con 0x55fdce6f4f40 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce707be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3cd9c6640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400bd10 tx=0x7fa3c400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a11802a3da81b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4005450 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa3c40055f0 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.178+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3b8013550 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3c4039050 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fdce775ee0 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3b8013550 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa3c4029210 con 0x55fdce6f4f40 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 20 client.15945 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce7763d0 0x55fdce7787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 4 client.15945 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdce661200 con 0x55fdce7763d0 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.179+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce7763d0 0x55fdce7787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.180+0000 7fa3b17fa640 10 client.15945 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.180+0000 7fa3cd1c5640 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce7763d0 0x55fdce7787f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ece5b9d4088d7c0 server_cookie=c7d16d22ed65d292 in_seq=0 out_seq=0 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3b17fa640 1 -- 172.21.15.32:0/248321925 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdce661200 con 0x55fdce7763d0 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3b17fa640 10 client.15945 handle_command_reply: tid=0 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 client.15945 shutdown 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 msgr2=0x55fdce704f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400bd10 tx=0x7fa3c400b390 comp rx=0 tx=0).stop 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 shutdown_connections 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fdce7763d0 0x55fdce7787f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdce6ff800 0x55fdce7054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3b17fa640 0 client.15945 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fdce6f4f40 0x55fdce704f70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 --2- 172.21.15.32:0/248321925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fdce6f4620 0x55fdce707be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 >> 172.21.15.32:0/248321925 conn(0x55fdce6f0b30 msgr2=0x55fdce6ef420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 shutdown_connections 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 1 -- 172.21.15.32:0/248321925 wait complete. 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.194+0000 7fa3d2a1a740 20 client.15945 trim_cache size 0 max 16384 2024-04-23T18:18:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:18:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:18:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:18:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 msgr2=0x7fa3b40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 0x7fa3b40a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00eef80 tx=0x7fa3c00095c0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 msgr2=0x7fa3c81a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa3c002ae70 tx=0x7fa3c002f5d0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 shutdown_connections 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3b40a3d90 0x7fa3b40a6250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c8112f60 0x7fa3c81a3670 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3c8074c10 0x7fa3c819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 --2- 172.21.15.32:0/3430501764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3c80742f0 0x7fa3c819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 >> 172.21.15.32:0/3430501764 conn(0x7fa3c80fda40 msgr2=0x7fa3c80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 shutdown_connections 2024-04-23T18:18:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:42.195+0000 7fa3cfc2a640 1 -- 172.21.15.32:0/3430501764 wait complete. 2024-04-23T18:18:42.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:42 smithi040 ceph-mon[32626]: pgmap v111: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 471 op/s 2024-04-23T18:18:42.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:42 smithi064 ceph-mon[33062]: pgmap v111: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 471 op/s 2024-04-23T18:18:43.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:42 smithi032 ceph-mon[29221]: pgmap v111: 97 pgs: 97 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 471 op/s 2024-04-23T18:18:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 195.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:44.809 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:44 smithi032 ceph-mon[29221]: pgmap v112: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:44.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:44 smithi040 ceph-mon[32626]: pgmap v112: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:44.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:44 smithi064 ceph-mon[33062]: pgmap v112: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:46.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:46 smithi040 ceph-mon[32626]: pgmap v113: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:46.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:46 smithi064 ceph-mon[33062]: pgmap v113: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:47.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:46 smithi032 ceph-mon[29221]: pgmap v113: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:47.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:47 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:47.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:47 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:47 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:48.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 200.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:48.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:48 smithi040 ceph-mon[32626]: pgmap v114: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:48.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:48 smithi064 ceph-mon[33062]: pgmap v114: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:49.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:48 smithi032 ceph-mon[29221]: pgmap v114: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 1.06k op/s 2024-04-23T18:18:50.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:50 smithi040 ceph-mon[32626]: pgmap v115: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:50.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:50 smithi064 ceph-mon[33062]: pgmap v115: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:51.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:50 smithi032 ceph-mon[29221]: pgmap v115: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-23T18:18:52.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:52 smithi040 ceph-mon[32626]: pgmap v116: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 610 op/s 2024-04-23T18:18:52.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:52 smithi064 ceph-mon[33062]: pgmap v116: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 610 op/s 2024-04-23T18:18:53.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:52 smithi032 ceph-mon[29221]: pgmap v116: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 610 op/s 2024-04-23T18:18:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 205.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:54.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:54 smithi040 ceph-mon[32626]: pgmap v117: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 614 op/s 2024-04-23T18:18:54.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:54 smithi064 ceph-mon[33062]: pgmap v117: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 614 op/s 2024-04-23T18:18:55.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:54 smithi032 ceph-mon[29221]: pgmap v117: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 614 op/s 2024-04-23T18:18:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 -- 172.21.15.32:0/3191812479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74102b80 msgr2=0x7efc74106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 --2- 172.21.15.32:0/3191812479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74102b80 0x7efc74106bc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009900 tx=0x7efc5c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 -- 172.21.15.32:0/3191812479 shutdown_connections 2024-04-23T18:18:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 --2- 172.21.15.32:0/3191812479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741072a0 0x7efc7410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 --2- 172.21.15.32:0/3191812479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74102b80 0x7efc74106bc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 --2- 172.21.15.32:0/3191812479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741021d0 0x7efc741025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 -- 172.21.15.32:0/3191812479 >> 172.21.15.32:0/3191812479 conn(0x7efc740fda30 msgr2=0x7efc740ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 -- 172.21.15.32:0/3191812479 shutdown_connections 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.294+0000 7efc7ab20640 1 -- 172.21.15.32:0/3191812479 wait complete. 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 Processor -- start 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 -- start start 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741021d0 0x7efc7419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 0x7efc7419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc741121c0 con 0x7efc74198690 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc74112040 con 0x7efc741072a0 2024-04-23T18:18:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc7ab20640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efc74112340 con 0x7efc741021d0 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54290/0 (socket says 172.21.15.32:54290) 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.295+0000 7efc790bd640 1 -- 172.21.15.32:0/1103074003 learned_addr learned my addr 172.21.15.32:0/1103074003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc788bc640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741021d0 0x7efc7419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc73fff640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 0x7efc7419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 -- 172.21.15.32:0/1103074003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741021d0 msgr2=0x7efc7419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741021d0 0x7efc7419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 -- 172.21.15.32:0/1103074003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 msgr2=0x7efc7419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 0x7efc7419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 -- 172.21.15.32:0/1103074003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc74199290 con 0x7efc74198690 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc73fff640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 0x7efc7419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc790bd640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00eaa0 tx=0x7efc6c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc71ffb640 1 -- 172.21.15.32:0/1103074003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00cdb0 con 0x7efc74198690 2024-04-23T18:18:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc741a5320 con 0x7efc74198690 2024-04-23T18:18:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc71ffb640 1 -- 172.21.15.32:0/1103074003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c005020 con 0x7efc74198690 2024-04-23T18:18:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.296+0000 7efc71ffb640 1 -- 172.21.15.32:0/1103074003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0116d0 con 0x7efc74198690 2024-04-23T18:18:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.297+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc741a56d0 con 0x7efc74198690 2024-04-23T18:18:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.299+0000 7efc71ffb640 1 -- 172.21.15.32:0/1103074003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7efc6c0051d0 con 0x7efc74198690 2024-04-23T18:18:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.299+0000 7efc7d838740 1 Processor -- start 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.299+0000 7efc7d838740 1 -- start start 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.299+0000 7efc71ffb640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 0x7efc600a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24488c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc73fff640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 0x7efc600a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc71ffb640 1 -- 172.21.15.32:0/1103074003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efc6c0d0910 con 0x7efc74198690 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ac23ad970 con 0x561ac2457c10 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561ac234a4a0 con 0x561ac24484e0 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc7d838740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ac23a9930 con 0x561ac2448e00 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc73fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54296/0 (socket says 172.21.15.32:54296) 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 -- 172.21.15.32:0/3451763923 learned_addr learned my addr 172.21.15.32:0/3451763923 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc788bc640 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 -- 172.21.15.32:0/3451763923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 msgr2=0x561ac24576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 -- 172.21.15.32:0/3451763923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 msgr2=0x561ac24488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 -- 172.21.15.32:0/3451763923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac20c0210 con 0x561ac2457c10 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc73fff640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 0x7efc600a61d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000c00 tx=0x7efc5c031580 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc790bd640 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0ce6b0 tx=0x7efc6c09c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17902d0596e9711 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc6affd640 1 -- 172.21.15.32:0/3451763923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c01f040 con 0x561ac2457c10 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc6affd640 1 -- 172.21.15.32:0/3451763923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0a5570 con 0x561ac2457c10 2024-04-23T18:18:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.300+0000 7efc6affd640 1 -- 172.21.15.32:0/3451763923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0870c0 con 0x561ac2457c10 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- 172.21.15.32:0/3451763923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 msgr2=0x561ac245a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0ce6b0 tx=0x7efc6c09c9c0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- 172.21.15.32:0/3451763923 shutdown_connections 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac245a000 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 --2- 172.21.15.32:0/3451763923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- 172.21.15.32:0/3451763923 >> 172.21.15.32:0/3451763923 conn(0x561ac2444b30 msgr2=0x561ac2444f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- 172.21.15.32:0/3451763923 shutdown_connections 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- 172.21.15.32:0/3451763923 wait complete. 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.301+0000 7efc7d838740 1 -- start start 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 Processor -- start 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac2457520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2457c10 0x561ac2454e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ac245a7e0 con 0x561ac2448e00 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561ac23480b0 con 0x561ac24484e0 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ac245a960 con 0x561ac2457c10 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc7d838740 10 client.0 _ll_register_callbacks cb 0x561ac2335d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc73fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc73fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54308/0 (socket says 172.21.15.32:54308) 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc73fff640 1 -- 172.21.15.32:0/3582853605 learned_addr learned my addr 172.21.15.32:0/3582853605 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc788bc640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac2457520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.302+0000 7efc790bd640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2457c10 0x561ac2454e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 -- 172.21.15.32:0/3582853605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2457c10 msgr2=0x561ac2454e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2457c10 0x561ac2454e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 -- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 msgr2=0x561ac2457520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac2457520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 -- 172.21.15.32:0/3582853605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac218c620 con 0x561ac2448e00 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc73fff640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009900 tx=0x7efc5c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed26b7c55d79f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc53fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c035050 con 0x561ac2448e00 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efc5c005450 con 0x561ac2448e00 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc6c0a5570 con 0x561ac2448e00 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc5c005650 con 0x561ac2448e00 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc7d838740 10 client.15957 fetch_fsmap learned FSMap version 29 2024-04-23T18:18:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ac24667b0 con 0x561ac2448e00 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.303+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0a5570 con 0x561ac2448e00 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7efc5c0036f0 con 0x561ac2448e00 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 10 client.15957 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 20 client.15957 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac2467c70 0x561ac246a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 4 client.15957 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ac23bb390 con 0x561ac2467c70 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc788bc640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac2467c70 0x561ac246a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc788bc640 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac2467c70 0x561ac246a060 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dd5e67fb4c99843 server_cookie=f47033fe130dce4a in_seq=0 out_seq=0 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.304+0000 7efc53fff640 10 client.15957 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc53fff640 1 -- 172.21.15.32:0/3582853605 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ac23bb390 con 0x561ac2467c70 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc53fff640 10 client.15957 handle_command_reply: tid=0 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 client.15957 shutdown 2024-04-23T18:18:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 msgr2=0x561ac2454880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009900 tx=0x7efc5c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 shutdown_connections 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac2467c70 0x561ac246a060 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2457c10 0x561ac2454e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2448e00 0x561ac2454880 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 --2- 172.21.15.32:0/3582853605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac2457520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc53fff640 0 client.15957 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.305+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 >> 172.21.15.32:0/3582853605 conn(0x561ac2444b30 msgr2=0x561ac244a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.306+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 shutdown_connections 2024-04-23T18:18:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.306+0000 7efc7d838740 1 -- 172.21.15.32:0/3582853605 wait complete. 2024-04-23T18:18:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.306+0000 7efc7d838740 20 client.15957 trim_cache size 0 max 16384 2024-04-23T18:18:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 Processor -- start 2024-04-23T18:18:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 -- start start 2024-04-23T18:18:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24567c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc741ae2d0 con 0x561ac2457c10 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc741abb10 con 0x561ac24484e0 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc7d838740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efc741a9980 con 0x561ac2448e00 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54316/0 (socket says 172.21.15.32:54316) 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 -- 172.21.15.32:0/1157104434 learned_addr learned my addr 172.21.15.32:0/1157104434 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc73fff640 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc788bc640 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 -- 172.21.15.32:0/1157104434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 msgr2=0x561ac24567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 -- 172.21.15.32:0/1157104434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 msgr2=0x561ac24636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24636e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.321+0000 7efc790bd640 1 -- 172.21.15.32:0/1157104434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac246a6e0 con 0x561ac2457c10 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc790bd640 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efc6c09c890 tx=0x7efc6c09c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4e6d61fca5836f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc6a7fc640 1 -- 172.21.15.32:0/1157104434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c01f070 con 0x561ac2457c10 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc6a7fc640 1 -- 172.21.15.32:0/1157104434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0b9070 con 0x561ac2457c10 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc6a7fc640 1 -- 172.21.15.32:0/1157104434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c087250 con 0x561ac2457c10 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 -- 172.21.15.32:0/1157104434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 msgr2=0x561ac246f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efc6c09c890 tx=0x7efc6c09c800 comp rx=0 tx=0).stop 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 -- 172.21.15.32:0/1157104434 shutdown_connections 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac2457c10 0x561ac246f970 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac24567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 --2- 172.21.15.32:0/1157104434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac24484e0 0x561ac24636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 -- 172.21.15.32:0/1157104434 >> 172.21.15.32:0/1157104434 conn(0x561ac2444b30 msgr2=0x561ac24560b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 -- 172.21.15.32:0/1157104434 shutdown_connections 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.322+0000 7efc7d838740 1 -- 172.21.15.32:0/1157104434 wait complete. 2024-04-23T18:18:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 -- start start 2024-04-23T18:18:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 Processor -- start 2024-04-23T18:18:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac2452150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac2457c10 0x561ac24cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc741a9fd0 con 0x561ac24484e0 2024-04-23T18:18:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc741ab440 con 0x561ac2457c10 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efc741aa6a0 con 0x561ac2448e00 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc7d838740 10 client.0 _ll_register_callbacks cb 0x7efc5c004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc788bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc788bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54326/0 (socket says 172.21.15.32:54326) 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.323+0000 7efc788bc640 1 -- 172.21.15.32:0/2860000969 learned_addr learned my addr 172.21.15.32:0/2860000969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc73fff640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac2452150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc790bd640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac2457c10 0x561ac24cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 -- 172.21.15.32:0/2860000969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 msgr2=0x561ac2452150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac2452150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 -- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac2457c10 msgr2=0x561ac24cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac2457c10 0x561ac24cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 -- 172.21.15.32:0/2860000969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac24cd720 con 0x561ac24484e0 2024-04-23T18:18:55.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc788bc640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efc64009fd0 tx=0x7efc64010ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82f8717efe10ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc53fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64023070 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efc6400fba0 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc6c0b9070 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc6402f050 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc7d838740 10 client.15963 fetch_fsmap learned FSMap version 29 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ac24ce9f0 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.324+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0b9070 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7efc6401e3f0 con 0x561ac24484e0 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 10 client.15963 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 10 client.15963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 20 client.15963 populate_metadata read hostname 'smithi032' 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac24cf150 0x561ac24d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 4 client.15963 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ac23b5200 con 0x561ac24cf150 2024-04-23T18:18:55.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc73fff640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac24cf150 0x561ac24d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc53fff640 10 client.15963 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.325+0000 7efc73fff640 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac24cf150 0x561ac24d1590 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdaa8e439a1dc1fe server_cookie=8d41da7687d62631 in_seq=0 out_seq=0 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc53fff640 1 -- 172.21.15.32:0/2860000969 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ac23b5200 con 0x561ac24cf150 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc53fff640 10 client.15963 handle_command_reply: tid=0 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 client.15963 shutdown 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 msgr2=0x561ac2451c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efc64009fd0 tx=0x7efc64010ba0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 shutdown_connections 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561ac24cf150 0x561ac24d1590 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ac2457c10 0x561ac24cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ac2448e00 0x561ac2452150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 --2- 172.21.15.32:0/2860000969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ac24484e0 0x561ac2451c10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc53fff640 0 client.15963 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 >> 172.21.15.32:0/2860000969 conn(0x561ac2444b30 msgr2=0x561ac244a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 shutdown_connections 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 1 -- 172.21.15.32:0/2860000969 wait complete. 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.336+0000 7efc7d838740 20 client.15963 trim_cache size 0 max 16384 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:18:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 msgr2=0x7efc600a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 0x7efc600a61d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000c00 tx=0x7efc5c031580 comp rx=0 tx=0).stop 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 msgr2=0x7efc74198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00eaa0 tx=0x7efc6c00ef60 comp rx=0 tx=0).stop 2024-04-23T18:18:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 shutdown_connections 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efc600a3d10 0x7efc600a61d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc74198690 0x7efc74198b10 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc741072a0 0x7efc7419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 --2- 172.21.15.32:0/1103074003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc741021d0 0x7efc7419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 >> 172.21.15.32:0/1103074003 conn(0x7efc740fda30 msgr2=0x7efc740feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 shutdown_connections 2024-04-23T18:18:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:18:55.337+0000 7efc7ab20640 1 -- 172.21.15.32:0/1103074003 wait complete. 2024-04-23T18:18:56.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:56 smithi040 ceph-mon[32626]: pgmap v118: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:18:56.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:56 smithi064 ceph-mon[33062]: pgmap v118: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:18:57.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:56 smithi032 ceph-mon[29221]: pgmap v118: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:18:58.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 210.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:18:58.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:18:58 smithi040 ceph-mon[32626]: pgmap v119: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:18:58.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:18:58 smithi064 ceph-mon[33062]: pgmap v119: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:18:59.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:18:58 smithi032 ceph-mon[29221]: pgmap v119: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 194 KiB/s wr, 17 op/s 2024-04-23T18:19:00.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:00 smithi040 ceph-mon[32626]: pgmap v120: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 KiB/s wr, 23 op/s 2024-04-23T18:19:00.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:00 smithi064 ceph-mon[33062]: pgmap v120: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 KiB/s wr, 23 op/s 2024-04-23T18:19:01.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:00 smithi032 ceph-mon[29221]: pgmap v120: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 KiB/s wr, 23 op/s 2024-04-23T18:19:02.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:02.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 -- 172.21.15.32:0/2830255758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102b50 msgr2=0x7fa364106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 --2- 172.21.15.32:0/2830255758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102b50 0x7fa364106b90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009900 tx=0x7fa34c02a410 comp rx=0 tx=0).stop 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 -- 172.21.15.32:0/2830255758 shutdown_connections 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 --2- 172.21.15.32:0/2830255758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa364107270 0x7fa36410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 --2- 172.21.15.32:0/2830255758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa364102b50 0x7fa364106b90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 --2- 172.21.15.32:0/2830255758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa3641021a0 0x7fa364102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 -- 172.21.15.32:0/2830255758 >> 172.21.15.32:0/2830255758 conn(0x7fa3640fda40 msgr2=0x7fa3640ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.168+0000 7fa369108640 1 -- 172.21.15.32:0/2830255758 shutdown_connections 2024-04-23T18:19:02.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.169+0000 7fa369108640 1 -- 172.21.15.32:0/2830255758 wait complete. 2024-04-23T18:19:02.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.169+0000 7fa369108640 1 Processor -- start 2024-04-23T18:19:02.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.169+0000 7fa369108640 1 -- start start 2024-04-23T18:19:02.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.169+0000 7fa369108640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3641021a0 0x7fa36419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.169+0000 7fa369108640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364107270 0x7fa36419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa369108640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa369108640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa364112200 con 0x7fa3641985e0 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa369108640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa364112080 con 0x7fa3641021a0 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa369108640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa364112380 con 0x7fa364107270 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33970/0 (socket says 172.21.15.32:33970) 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 -- 172.21.15.32:0/2075195289 learned_addr learned my addr 172.21.15.32:0/2075195289 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa35bfff640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364107270 0x7fa36419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa362d9d640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3641021a0 0x7fa36419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 -- 172.21.15.32:0/2075195289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364107270 msgr2=0x7fa36419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364107270 0x7fa36419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 -- 172.21.15.32:0/2075195289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3641021a0 msgr2=0x7fa36419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3641021a0 0x7fa36419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 -- 172.21.15.32:0/2075195289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa364199100 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa36359e640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00eaa0 tx=0x7fa35c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c00cdb0 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa35c005020 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3641993f0 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.170+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0116d0 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.171+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3641a55f0 con 0x7fa3641985e0 2024-04-23T18:19:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.171+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa364069250 con 0x7fa3641985e0 2024-04-23T18:19:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.173+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa35c0051d0 con 0x7fa3641985e0 2024-04-23T18:19:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.173+0000 7fa360d99640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 0x7fa3500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.174+0000 7fa362d9d640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 0x7fa3500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.174+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa35c0d0800 con 0x7fa3641985e0 2024-04-23T18:19:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.174+0000 7fa362d9d640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 0x7fa3500a6260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa354005fd0 tx=0x7fa35400b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.178+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa35c016030 con 0x7fa3641985e0 2024-04-23T18:19:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.406+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa36410a4a0 con 0x7fa3641985e0 2024-04-23T18:19:02.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.407+0000 7fa360d99640 1 -- 172.21.15.32:0/2075195289 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11855 (secure 0 0 0) 0x7fa35c09c8a0 con 0x7fa3641985e0 2024-04-23T18:19:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:19:02.400 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:18:45.862685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:19:02.401 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-23T18:19:02.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 msgr2=0x7fa3500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 0x7fa3500a6260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa354005fd0 tx=0x7fa35400b390 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 msgr2=0x7fa364198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00eaa0 tx=0x7fa35c00ef60 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 shutdown_connections 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa3500a3da0 0x7fa3500a6260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3641985e0 0x7fa364198a90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa364107270 0x7fa36419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.408+0000 7fa369108640 1 --2- 172.21.15.32:0/2075195289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3641021a0 0x7fa36419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.409+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 >> 172.21.15.32:0/2075195289 conn(0x7fa3640fda40 msgr2=0x7fa3640fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.409+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 shutdown_connections 2024-04-23T18:19:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.409+0000 7fa369108640 1 -- 172.21.15.32:0/2075195289 wait complete. 2024-04-23T18:19:02.413 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id d7c1a45b --release --await' 2024-04-23T18:19:02.414 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id d7c1a45b --release --await 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 -- 172.21.15.32:0/1896782432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc10103b70 msgr2=0x7fcc1010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/1896782432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc10103b70 0x7fcc1010bfc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009900 tx=0x7fcc0402a410 comp rx=0 tx=0).stop 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 -- 172.21.15.32:0/1896782432 shutdown_connections 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/1896782432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1010c6d0 0x7fcc1010eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/1896782432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc10103b70 0x7fcc1010bfc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/1896782432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 0x7fcc101035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 -- 172.21.15.32:0/1896782432 >> 172.21.15.32:0/1896782432 conn(0x7fcc10077f70 msgr2=0x7fcc10078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 -- 172.21.15.32:0/1896782432 shutdown_connections 2024-04-23T18:19:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.529+0000 7fcc15b9b640 1 -- 172.21.15.32:0/1896782432 wait complete. 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 Processor -- start 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 -- start start 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 0x7fcc1019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc101987e0 0x7fcc10198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc10112350 con 0x7fcc1010c6d0 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101121d0 con 0x7fcc101987e0 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc15b9b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101124d0 con 0x7fcc101031c0 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33992/0 (socket says 172.21.15.32:33992) 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 -- 172.21.15.32:0/4168058212 learned_addr learned my addr 172.21.15.32:0/4168058212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 0x7fcc1019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 -- 172.21.15.32:0/4168058212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 msgr2=0x7fcc1019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 0x7fcc1019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 -- 172.21.15.32:0/4168058212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc101987e0 msgr2=0x7fcc10198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc101987e0 0x7fcc10198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.530+0000 7fcc0effd640 1 -- 172.21.15.32:0/4168058212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc10199370 con 0x7fcc1010c6d0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc0effd640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcc04006010 tx=0x7fcc04004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc0cff9640 1 -- 172.21.15.32:0/4168058212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0403f030 con 0x7fcc1010c6d0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc0cff9640 1 -- 172.21.15.32:0/4168058212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc04003aa0 con 0x7fcc1010c6d0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc10199600 con 0x7fcc1010c6d0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc0cff9640 1 -- 172.21.15.32:0/4168058212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04035070 con 0x7fcc1010c6d0 2024-04-23T18:19:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.531+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc101a57b0 con 0x7fcc1010c6d0 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc0cff9640 1 -- 172.21.15.32:0/4168058212 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcc04042030 con 0x7fcc1010c6d0 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 Processor -- start 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 -- start start 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 0x564d7bd235a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd32350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564d7bc886d0 con 0x564d7bd32890 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564d7bc251a0 con 0x564d7bd23b70 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.533+0000 7fcc18993740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564d7bc84690 con 0x564d7bd231c0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0cff9640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 0x7fcbf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 0x7fcbf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34004/0 (socket says 172.21.15.32:34004) 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 -- 172.21.15.32:0/2255100154 learned_addr learned my addr 172.21.15.32:0/2255100154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 0x564d7bd235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd32350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0cff9640 1 -- 172.21.15.32:0/4168058212 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc040f0cc0 con 0x7fcc1010c6d0 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 -- 172.21.15.32:0/2255100154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 msgr2=0x564d7bd235a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 0x564d7bd235a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 -- 172.21.15.32:0/2255100154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 msgr2=0x564d7bd32350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd32350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 -- 172.21.15.32:0/2255100154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d7bc2f5b0 con 0x564d7bd32890 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 0x564d7bd235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 0x7fcbf80a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009370 tx=0x7fcc08009bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c53f0 tx=0x7fcc04000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b516a1eeed7ddd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcbf5ffb640 1 -- 172.21.15.32:0/2255100154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04049070 con 0x564d7bd32890 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcbf5ffb640 1 -- 172.21.15.32:0/2255100154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc040c5920 con 0x564d7bd32890 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.534+0000 7fcbf5ffb640 1 -- 172.21.15.32:0/2255100154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b9320 con 0x564d7bd32890 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- 172.21.15.32:0/2255100154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 msgr2=0x564d7bd34c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c53f0 tx=0x7fcc04000bc0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- 172.21.15.32:0/2255100154 shutdown_connections 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd32890 0x564d7bd34c80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd32350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 --2- 172.21.15.32:0/2255100154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd231c0 0x564d7bd235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- 172.21.15.32:0/2255100154 >> 172.21.15.32:0/2255100154 conn(0x564d7bd1f7b0 msgr2=0x564d7bd1fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- 172.21.15.32:0/2255100154 shutdown_connections 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- 172.21.15.32:0/2255100154 wait complete. 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.535+0000 7fcc18993740 1 -- start start 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 Processor -- start 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd2f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd2fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564d7bd35410 con 0x564d7bd231c0 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564d7bc22b20 con 0x564d7bd23b70 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564d7bd35590 con 0x564d7bd32890 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc18993740 10 client.0 _ll_register_callbacks cb 0x564d7bc22530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34020/0 (socket says 172.21.15.32:34020) 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2434345598 learned_addr learned my addr 172.21.15.32:0/2434345598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd2f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd2fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2434345598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 msgr2=0x564d7bd2fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd2fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 msgr2=0x564d7bd2f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd2f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.536+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2434345598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d7b99a920 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcc000136e0 tx=0x7fcc00013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff3e9ba6d1d5805 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011dd0 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc0001b3e0 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc040c5920 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc0001b5e0 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 10 client.15981 fetch_fsmap learned FSMap version 29 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564d7ba66d30 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc040c5920 con 0x564d7bd231c0 2024-04-23T18:19:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fcc0001a070 con 0x564d7bd231c0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 10 client.15981 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 10 client.15981 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 20 client.15981 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bd427d0 0x564d7bd44c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 4 client.15981 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.537+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d7bc46500 con 0x564d7bd427d0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.538+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bd427d0 0x564d7bd44c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.538+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bd427d0 0x564d7bd44c70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d53edb85ce8471f6 server_cookie=933a01b9bda91054 in_seq=0 out_seq=0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.538+0000 7fcbeeffd640 10 client.15981 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2434345598 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d7bc46500 con 0x564d7bd427d0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcbeeffd640 10 client.15981 handle_command_reply: tid=0 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 client.15981 shutdown 2024-04-23T18:19:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 msgr2=0x564d7bd32250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcc000136e0 tx=0x7fcc00013ba0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 shutdown_connections 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bd427d0 0x564d7bd44c70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd2fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd2f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcbeeffd640 0 client.15981 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 --2- 172.21.15.32:0/2434345598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd32250 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 >> 172.21.15.32:0/2434345598 conn(0x564d7bd1f7b0 msgr2=0x564d7bd25620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 shutdown_connections 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 1 -- 172.21.15.32:0/2434345598 wait complete. 2024-04-23T18:19:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.539+0000 7fcc18993740 20 client.15981 trim_cache size 0 max 16384 2024-04-23T18:19:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 Processor -- start 2024-04-23T18:19:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 -- start start 2024-04-23T18:19:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd231c0 0x564d7bd3e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd4a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101ae270 con 0x564d7bd23b70 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101abab0 con 0x564d7bd231c0 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc18993740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101a9920 con 0x564d7bd32890 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.556+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34024/0 (socket says 172.21.15.32:34024) 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 -- 172.21.15.32:0/3005704429 learned_addr learned my addr 172.21.15.32:0/3005704429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd4a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 -- 172.21.15.32:0/3005704429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 msgr2=0x564d7bd4a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd4a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd231c0 0x564d7bd3e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 -- 172.21.15.32:0/3005704429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd231c0 msgr2=0x564d7bd3e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd231c0 0x564d7bd3e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 -- 172.21.15.32:0/3005704429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d7bd30040 con 0x564d7bd23b70 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcc0effd640 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcc040b9680 tx=0x7fcc040e4040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8030e082112dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcbf57fa640 1 -- 172.21.15.32:0/3005704429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04049040 con 0x564d7bd23b70 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcbf57fa640 1 -- 172.21.15.32:0/3005704429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc040c5920 con 0x564d7bd23b70 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.557+0000 7fcbf57fa640 1 -- 172.21.15.32:0/3005704429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b9320 con 0x564d7bd23b70 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- 172.21.15.32:0/3005704429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 msgr2=0x564d7bd31d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcc040b9680 tx=0x7fcc040e4040 comp rx=0 tx=0).stop 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- 172.21.15.32:0/3005704429 shutdown_connections 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd4a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd23b70 0x564d7bd31d80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 --2- 172.21.15.32:0/3005704429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd231c0 0x564d7bd3e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- 172.21.15.32:0/3005704429 >> 172.21.15.32:0/3005704429 conn(0x564d7bd1f7b0 msgr2=0x564d7bd24b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- 172.21.15.32:0/3005704429 shutdown_connections 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- 172.21.15.32:0/3005704429 wait complete. 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 -- start start 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.558+0000 7fcc18993740 1 Processor -- start 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd332d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd33810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101a9f70 con 0x564d7bd231c0 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101ab3e0 con 0x564d7bd23b70 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101aa640 con 0x564d7bd32890 2024-04-23T18:19:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc18993740 10 client.0 _ll_register_callbacks cb 0x7fcc00009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34036/0 (socket says 172.21.15.32:34036) 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2021920604 learned_addr learned my addr 172.21.15.32:0/2021920604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0ffff640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd33810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2021920604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 msgr2=0x564d7bd33810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd33810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 msgr2=0x564d7bd332d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd332d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 -- 172.21.15.32:0/2021920604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d7bd33f00 con 0x564d7bd231c0 2024-04-23T18:19:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd332d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcc0f7fe640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000c00 tx=0x7fcc00011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61efb2f9f7e3229 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.559+0000 7fcbeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:19:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00029070 con 0x564d7bd231c0 2024-04-23T18:19:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc0000e5e0 con 0x564d7bd231c0 2024-04-23T18:19:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc040c5920 con 0x564d7bd231c0 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc00035050 con 0x564d7bd231c0 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcc18993740 10 client.15987 fetch_fsmap learned FSMap version 29 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564d7bda8df0 con 0x564d7bd231c0 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.561+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc040c5920 con 0x564d7bd231c0 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fcc0001a040 con 0x564d7bd231c0 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 10 client.15987 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 20 client.15987 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bda94f0 0x564d7bdab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 4 client.15987 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d7c1a45b", "release": true, "await": true}] 2024-04-23T18:19:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d7c1a45b", "release": true, "await": true}) -- 0x564d7bc960f0 con 0x564d7bda94f0 2024-04-23T18:19:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bda94f0 0x564d7bdab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcc0effd640 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bda94f0 0x564d7bdab910 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c18e59eecb2e6178 server_cookie=830c10aa0e06211b in_seq=0 out_seq=0 2024-04-23T18:19:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.562+0000 7fcbeeffd640 10 client.15987 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:02.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcbeeffd640 1 -- 172.21.15.32:0/2021920604 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x564d7bc960f0 con 0x564d7bda94f0 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcbeeffd640 10 client.15987 handle_command_reply: tid=0 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 client.15987 shutdown 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 msgr2=0x564d7bd3aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000c00 tx=0x7fcc00011e00 comp rx=0 tx=0).stop 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 shutdown_connections 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564d7bda94f0 0x564d7bdab910 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d7bd32890 0x564d7bd33810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d7bd23b70 0x564d7bd332d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcbeeffd640 0 client.15987 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 --2- 172.21.15.32:0/2021920604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564d7bd231c0 0x564d7bd3aac0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 >> 172.21.15.32:0/2021920604 conn(0x564d7bd1f7b0 msgr2=0x564d7bd252e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 shutdown_connections 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.570+0000 7fcc18993740 1 -- 172.21.15.32:0/2021920604 wait complete. 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc18993740 20 client.15987 trim_cache size 0 max 16384 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 29, 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 12, 2024-04-23T18:19:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "d7c1a45b": { 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 12, 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 94.1, 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:19:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:19:02.568 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 msgr2=0x7fcbf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 0x7fcbf80a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009370 tx=0x7fcc08009bc0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 msgr2=0x7fcc1019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcc04006010 tx=0x7fcc04004060 comp rx=0 tx=0).stop 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 shutdown_connections 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc101987e0 0x7fcc10198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcbf80a3d20 0x7fcbf80a61e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc1010c6d0 0x7fcc1019ebb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 --2- 172.21.15.32:0/4168058212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc101031c0 0x7fcc1019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 >> 172.21.15.32:0/4168058212 conn(0x7fcc10077f70 msgr2=0x7fcc100ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 shutdown_connections 2024-04-23T18:19:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:19:02.571+0000 7fcc15b9b640 1 -- 172.21.15.32:0/4168058212 wait complete. 2024-04-23T18:19:02.576 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd7c1a45b', seconds elapsed: 35.7 2024-04-23T18:19:02.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:02 smithi032 ceph-mon[29221]: pgmap v121: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-04-23T18:19:02.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2075195289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:02.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:02 smithi040 ceph-mon[32626]: pgmap v121: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-04-23T18:19:02.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2075195289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:02.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:02 smithi064 ceph-mon[33062]: pgmap v121: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-04-23T18:19:02.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2075195289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 215.0 s, 724.0 tps, lat 998.621 ms stddev 5722.814 2024-04-23T18:19:04.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:04 smithi040 ceph-mon[32626]: pgmap v122: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 478 KiB/s wr, 73 op/s 2024-04-23T18:19:04.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:04 smithi064 ceph-mon[33062]: pgmap v122: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 478 KiB/s wr, 73 op/s 2024-04-23T18:19:05.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:04 smithi032 ceph-mon[29221]: pgmap v122: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 478 KiB/s wr, 73 op/s 2024-04-23T18:19:06.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:06 smithi040 ceph-mon[32626]: pgmap v123: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:06.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:06 smithi064 ceph-mon[33062]: pgmap v123: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:07.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:06 smithi032 ceph-mon[29221]: pgmap v123: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1467451979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e4108f60 msgr2=0x7fc3e4115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1467451979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e4108f60 0x7fc3e4115420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009ca0 tx=0x7fc3dc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1467451979 shutdown_connections 2024-04-23T18:19:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1467451979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e4108f60 0x7fc3e4115420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1467451979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e41085a0 0x7fc3e4108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1467451979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1467451979 >> 172.21.15.32:0/1467451979 conn(0x7fc3e4076500 msgr2=0x7fc3e4078920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1467451979 shutdown_connections 2024-04-23T18:19:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.464+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1467451979 wait complete. 2024-04-23T18:19:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.465+0000 7fc3e9c31640 1 Processor -- start 2024-04-23T18:19:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.465+0000 7fc3e9c31640 1 -- start start 2024-04-23T18:19:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e41085a0 0x7fc3e41a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e4108f60 0x7fc3e41a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4118850 con 0x7fc3e41085a0 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41186d0 con 0x7fc3e410e660 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e9c31640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41189d0 con 0x7fc3e4108f60 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36068/0 (socket says 172.21.15.32:36068) 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e3fff640 1 -- 172.21.15.32:0/1759990898 learned_addr learned my addr 172.21.15.32:0/1759990898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e41085a0 0x7fc3e41a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.466+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e4108f60 0x7fc3e41a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 -- 172.21.15.32:0/1759990898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e4108f60 msgr2=0x7fc3e41a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e4108f60 0x7fc3e41a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 -- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e41085a0 msgr2=0x7fc3e41a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e41085a0 0x7fc3e41a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 -- 172.21.15.32:0/1759990898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e419f2b0 con 0x7fc3e410e660 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0319f0 tx=0x7fc3dc031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e0ff9640 1 -- 172.21.15.32:0/1759990898 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc003b70 con 0x7fc3e410e660 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e419f540 con 0x7fc3e410e660 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e0ff9640 1 -- 172.21.15.32:0/1759990898 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3dc003d10 con 0x7fc3e410e660 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e0ff9640 1 -- 172.21.15.32:0/1759990898 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc02fa10 con 0x7fc3e410e660 2024-04-23T18:19:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.467+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e41ab7f0 con 0x7fc3e410e660 2024-04-23T18:19:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.471+0000 7fc3eca22740 1 Processor -- start 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.471+0000 7fc3e0ff9640 1 -- 172.21.15.32:0/1759990898 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc3dc02fbb0 con 0x7fc3e410e660 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 -- start start 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e0ff9640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 0x7fc3cc0a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9f620 0x555f17d9fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17dacbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555f17d04970 con 0x555f17d9ff40 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555f17ca14a0 con 0x555f17daa800 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3eca22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555f17d00930 con 0x555f17d9f620 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e0ff9640 1 -- 172.21.15.32:0/1759990898 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc3dc0f1bd0 con 0x7fc3e410e660 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 0x7fc3cc0a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17dacbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44402/0 (socket says 172.21.15.32:44402) 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/2891171938 learned_addr learned my addr 172.21.15.32:0/2891171938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.472+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9f620 0x555f17d9fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/2891171938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9f620 msgr2=0x555f17d9fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9f620 0x555f17d9fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/2891171938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 msgr2=0x555f17dacbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17dacbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/2891171938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f17a17210 con 0x555f17d9ff40 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 0x7fc3cc0a6220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8013570 tx=0x7fc3d8013970 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.473+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8013e70 tx=0x7fc3d8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10197519ae483809 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.474+0000 7fc3c9ffb640 1 -- 172.21.15.32:0/2891171938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800fe30 con 0x555f17d9ff40 2024-04-23T18:19:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.474+0000 7fc3c9ffb640 1 -- 172.21.15.32:0/2891171938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3d8024030 con 0x555f17d9ff40 2024-04-23T18:19:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.474+0000 7fc3c9ffb640 1 -- 172.21.15.32:0/2891171938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8010b10 con 0x555f17d9ff40 2024-04-23T18:19:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 -- 172.21.15.32:0/2891171938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 msgr2=0x555f17daa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8013e70 tx=0x7fc3d8020000 comp rx=0 tx=0).stop 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 -- 172.21.15.32:0/2891171938 shutdown_connections 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17dacbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9ff40 0x555f17daa2c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2891171938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9f620 0x555f17d9fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 -- 172.21.15.32:0/2891171938 >> 172.21.15.32:0/2891171938 conn(0x555f17d9bb30 msgr2=0x555f17d9bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 -- 172.21.15.32:0/2891171938 shutdown_connections 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.475+0000 7fc3eca22740 1 -- 172.21.15.32:0/2891171938 wait complete. 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.476+0000 7fc3eca22740 1 -- start start 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.476+0000 7fc3eca22740 1 Processor -- start 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.476+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17daa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.476+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17da79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36090/0 (socket says 172.21.15.32:36090) 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/3102021619 learned_addr learned my addr 172.21.15.32:0/3102021619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17daa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555f17dad3d0 con 0x555f17d9f620 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555f17c9f0b0 con 0x555f17d9ff40 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555f17dad550 con 0x555f17daa800 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17da79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3eca22740 10 client.0 _ll_register_callbacks cb 0x555f17c8cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/3102021619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 msgr2=0x555f17da79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17da79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/3102021619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 msgr2=0x555f17daa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17daa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e2ffd640 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f17ae3620 con 0x555f17d9ff40 2024-04-23T18:19:08.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.477+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17da79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.478+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009a40 tx=0x7fc3d800f5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59003418d4e2c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.478+0000 7fc3c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.478+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8010a90 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.478+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc3d8010c30 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.478+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d8024030 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.479+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3d8010e30 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.479+0000 7fc3eca22740 10 client.25634 fetch_fsmap learned FSMap version 29 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.479+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f17db9300 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3d80054f0 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc3d801f070 con 0x555f17d9ff40 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 10 client.25634 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 20 client.25634 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17dbcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 4 client.25634 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc3d8024030 con 0x555f17dba710 2024-04-23T18:19:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.480+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17dbcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.481+0000 7fc3c2ffd640 10 client.25634 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.481+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17dbcb00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f551e19f5023a461 server_cookie=f9134deee079a5b6 in_seq=0 out_seq=0 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.482+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/3102021619 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc3d8024030 con 0x555f17dba710 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.482+0000 7fc3c2ffd640 10 client.25634 handle_command_reply: tid=0 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.482+0000 7fc3eca22740 1 client.25634 shutdown 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.482+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 msgr2=0x555f17da7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.482+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009a40 tx=0x7fc3d800f5c0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 shutdown_connections 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17dbcb00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17da79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3c2ffd640 0 client.25634 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17da7450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 --2- 172.21.15.32:0/3102021619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17daa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 >> 172.21.15.32:0/3102021619 conn(0x555f17d9bb30 msgr2=0x555f17da18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 shutdown_connections 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 1 -- 172.21.15.32:0/3102021619 wait complete. 2024-04-23T18:19:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.483+0000 7fc3eca22740 20 client.25634 trim_cache size 0 max 16384 2024-04-23T18:19:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.509+0000 7fc3eca22740 1 Processor -- start 2024-04-23T18:19:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.509+0000 7fc3eca22740 1 -- start start 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17dc2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17dc2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b6ac0 con 0x555f17d9f620 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b4300 con 0x555f17d9ff40 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3eca22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b2150 con 0x555f17daa800 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44424/0 (socket says 172.21.15.32:44424) 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 -- 172.21.15.32:0/2911871819 learned_addr learned my addr 172.21.15.32:0/2911871819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17dc2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17dc2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 -- 172.21.15.32:0/2911871819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 msgr2=0x555f17dc2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17dc2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 -- 172.21.15.32:0/2911871819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 msgr2=0x555f17dc2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17dc2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.510+0000 7fc3e37fe640 1 -- 172.21.15.32:0/2911871819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f17da7d20 con 0x555f17d9f620 2024-04-23T18:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.511+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4005fa0 tx=0x7fc3d40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb3d764bd376fea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.512+0000 7fc3c97fa640 1 -- 172.21.15.32:0/2911871819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d401d6f0 con 0x555f17d9f620 2024-04-23T18:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.512+0000 7fc3c97fa640 1 -- 172.21.15.32:0/2911871819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3d401d890 con 0x555f17d9f620 2024-04-23T18:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.512+0000 7fc3c97fa640 1 -- 172.21.15.32:0/2911871819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4003df0 con 0x555f17d9f620 2024-04-23T18:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- 172.21.15.32:0/2911871819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 msgr2=0x555f17db6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4005fa0 tx=0x7fc3d40118d0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- 172.21.15.32:0/2911871819 shutdown_connections 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17daa800 0x555f17dc2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17d9ff40 0x555f17dc2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 --2- 172.21.15.32:0/2911871819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db6290 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- 172.21.15.32:0/2911871819 >> 172.21.15.32:0/2911871819 conn(0x555f17d9bb30 msgr2=0x555f17d9acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- 172.21.15.32:0/2911871819 shutdown_connections 2024-04-23T18:19:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- 172.21.15.32:0/2911871819 wait complete. 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.513+0000 7fc3eca22740 1 -- start start 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 Processor -- start 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9ff40 0x555f17dafde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17db0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44430/0 (socket says 172.21.15.32:44430) 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3e37fe640 1 -- 172.21.15.32:0/4120738975 learned_addr learned my addr 172.21.15.32:0/4120738975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b27c0 con 0x555f17d9f620 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.514+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b3c30 con 0x555f17daa800 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e2ffd640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9ff40 0x555f17dafde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41b2e90 con 0x555f17d9ff40 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17db0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3eca22740 10 client.0 _ll_register_callbacks cb 0x7fc3d800f750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 -- 172.21.15.32:0/4120738975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9ff40 msgr2=0x555f17dafde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9ff40 0x555f17dafde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 -- 172.21.15.32:0/4120738975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 msgr2=0x555f17db2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3e3fff640 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f17dbce70 con 0x555f17daa800 2024-04-23T18:19:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.515+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d401d890 con 0x555f17daa800 2024-04-23T18:19:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.516+0000 7fc3e3fff640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17db0320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ef480 tx=0x7fc3dc0b9850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7c4a4de1f6209d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.516+0000 7fc3c2ffd640 10 client.25640 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.516+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/4120738975 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0bdcd0 con 0x555f17daa800 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/4120738975 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc3dc0c64b0 con 0x555f17daa800 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/4120738975 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3dc0c66f0 con 0x555f17daa800 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 10 client.25640 fetch_fsmap learned FSMap version 29 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=29}) -- 0x555f17e205f0 con 0x555f17daa800 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/4120738975 <== mon.2 v2:172.21.15.40:3300/0 4 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc3dc0de0b0 con 0x555f17daa800 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 10 client.25640 fetch_fsmap finished waiting for FSMap version 29 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 10 client.25640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 20 client.25640 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17e22cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 4 client.25640 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.517+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f17d12390 con 0x555f17dba710 2024-04-23T18:19:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.518+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17e22cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.518+0000 7fc3e37fe640 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17e22cb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3cde81bf74ec56e server_cookie=ffb8bea374f89e34 in_seq=0 out_seq=0 2024-04-23T18:19:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.518+0000 7fc3c2ffd640 10 client.25640 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.555+0000 7fc3c2ffd640 1 -- 172.21.15.32:0/4120738975 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f17d12390 con 0x555f17dba710 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.555+0000 7fc3c2ffd640 10 client.25640 handle_command_reply: tid=0 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.555+0000 7fc3eca22740 1 client.25640 shutdown 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.556+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 msgr2=0x555f17db0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.556+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17db0320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ef480 tx=0x7fc3dc0b9850 comp rx=0 tx=0).stop 2024-04-23T18:19:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 shutdown_connections 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555f17dba710 0x555f17e22cb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f17daa800 0x555f17db0320 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f17d9ff40 0x555f17dafde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 --2- 172.21.15.32:0/4120738975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f17d9f620 0x555f17db2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.557+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 >> 172.21.15.32:0/4120738975 conn(0x555f17d9bb30 msgr2=0x555f17db5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.558+0000 7fc3c2ffd640 0 client.25640 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.560+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 shutdown_connections 2024-04-23T18:19:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.561+0000 7fc3eca22740 1 -- 172.21.15.32:0/4120738975 wait complete. 2024-04-23T18:19:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.561+0000 7fc3eca22740 20 client.25640 trim_cache size 0 max 16384 2024-04-23T18:19:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:19:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:19:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:19:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.567+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 msgr2=0x7fc3cc0a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.567+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 0x7fc3cc0a6220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8013570 tx=0x7fc3d8013970 comp rx=0 tx=0).stop 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.567+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 msgr2=0x7fc3e419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.567+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0319f0 tx=0x7fc3dc031a20 comp rx=0 tx=0).stop 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 shutdown_connections 2024-04-23T18:19:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e410e660 0x7fc3e419eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e4108f60 0x7fc3e41a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc3cc0a3d60 0x7fc3cc0a6220 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 --2- 172.21.15.32:0/1759990898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3e41085a0 0x7fc3e41a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 >> 172.21.15.32:0/1759990898 conn(0x7fc3e4076500 msgr2=0x7fc3e4077d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 shutdown_connections 2024-04-23T18:19:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:08.568+0000 7fc3e9c31640 1 -- 172.21.15.32:0/1759990898 wait complete. 2024-04-23T18:19:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 220.0 s, 3024.6 tps, lat 33.012 ms stddev 33.110 2024-04-23T18:19:08.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:08 smithi040 ceph-mon[32626]: pgmap v124: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:08.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:08 smithi064 ceph-mon[33062]: pgmap v124: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:09.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:08 smithi032 ceph-mon[29221]: pgmap v124: 97 pgs: 97 active+clean; 2.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 476 KiB/s wr, 70 op/s 2024-04-23T18:19:10.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:10 smithi040 ceph-mon[32626]: pgmap v125: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 369 op/s 2024-04-23T18:19:10.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:10 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:10.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:10 smithi064 ceph-mon[33062]: pgmap v125: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 369 op/s 2024-04-23T18:19:10.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:10 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:10 smithi032 ceph-mon[29221]: pgmap v125: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 369 op/s 2024-04-23T18:19:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:10 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:12.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:12 smithi040 ceph-mon[32626]: pgmap v126: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 363 op/s 2024-04-23T18:19:12.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:12 smithi064 ceph-mon[33062]: pgmap v126: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 363 op/s 2024-04-23T18:19:13.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:12 smithi032 ceph-mon[29221]: pgmap v126: 97 pgs: 97 active+clean; 2.4 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 363 op/s 2024-04-23T18:19:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 225.0 s, 3651.8 tps, lat 27.307 ms stddev 28.649 2024-04-23T18:19:14.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:14 smithi040 ceph-mon[32626]: pgmap v127: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 7.9 MiB/s wr, 680 op/s 2024-04-23T18:19:14.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:14 smithi064 ceph-mon[33062]: pgmap v127: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 7.9 MiB/s wr, 680 op/s 2024-04-23T18:19:15.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:14 smithi032 ceph-mon[29221]: pgmap v127: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 7.9 MiB/s wr, 680 op/s 2024-04-23T18:19:16.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:16 smithi040 ceph-mon[32626]: pgmap v128: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:16.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:16 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:16.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:16 smithi064 ceph-mon[33062]: pgmap v128: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:16.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:16 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:17.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:16 smithi032 ceph-mon[29221]: pgmap v128: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:17.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:16 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:17.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:17 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:17.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:17.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:17.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:17 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:19:17.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:17 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:17.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:17.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:17.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:17 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:19:18.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:17 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:18.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:18.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:18.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:17 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:19:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 230.0 s, 3562.0 tps, lat 28.180 ms stddev 33.422 2024-04-23T18:19:18.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:18 smithi040 ceph-mon[32626]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:18.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:18 smithi064 ceph-mon[33062]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:19.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:18 smithi032 ceph-mon[29221]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.5 MiB/s wr, 616 op/s 2024-04-23T18:19:20.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:20 smithi040 ceph-mon[32626]: pgmap v130: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 938 op/s 2024-04-23T18:19:20.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:20 smithi064 ceph-mon[33062]: pgmap v130: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 938 op/s 2024-04-23T18:19:21.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:20 smithi032 ceph-mon[29221]: pgmap v130: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 938 op/s 2024-04-23T18:19:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 -- 172.21.15.32:0/671658540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad8101fe0 msgr2=0x7f4ad81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 --2- 172.21.15.32:0/671658540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81023c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009900 tx=0x7f4acc02a410 comp rx=0 tx=0).stop 2024-04-23T18:19:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 -- 172.21.15.32:0/671658540 shutdown_connections 2024-04-23T18:19:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 --2- 172.21.15.32:0/671658540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad81070b0 0x7f4ad810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 --2- 172.21.15.32:0/671658540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad8102990 0x7f4ad81069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 --2- 172.21.15.32:0/671658540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81023c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.713+0000 7f4ad759e640 1 -- 172.21.15.32:0/671658540 >> 172.21.15.32:0/671658540 conn(0x7f4ad80fd900 msgr2=0x7f4ad80ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.714+0000 7f4ad759e640 1 -- 172.21.15.32:0/671658540 shutdown_connections 2024-04-23T18:19:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.714+0000 7f4ad759e640 1 -- 172.21.15.32:0/671658540 wait complete. 2024-04-23T18:19:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.715+0000 7f4ad759e640 1 Processor -- start 2024-04-23T18:19:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.716+0000 7f4ad759e640 1 -- start start 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.716+0000 7f4ad759e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.716+0000 7f4ad759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad81070b0 0x7f4ad81a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.716+0000 7f4ad759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad810ddd0 0x7f4ad810e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad759e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad8112040 con 0x7f4ad810ddd0 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad759e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad8111ec0 con 0x7f4ad81070b0 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad759e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad81121c0 con 0x7f4ad8101fe0 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad81070b0 0x7f4ad81a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36792/0 (socket says 172.21.15.32:36792) 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 -- 172.21.15.32:0/2025659276 learned_addr learned my addr 172.21.15.32:0/2025659276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad6d9d640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad810ddd0 0x7f4ad810e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 -- 172.21.15.32:0/2025659276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad81070b0 msgr2=0x7f4ad81a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad81070b0 0x7f4ad81a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 -- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad810ddd0 msgr2=0x7f4ad810e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad810ddd0 0x7f4ad810e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 -- 172.21.15.32:0/2025659276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad810e980 con 0x7f4ad8101fe0 2024-04-23T18:19:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.717+0000 7f4ad659c640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4acc031440 tx=0x7f4acc003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.718+0000 7f4ac77fe640 1 -- 172.21.15.32:0/2025659276 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc03f070 con 0x7f4ad8101fe0 2024-04-23T18:19:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.718+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad810ec10 con 0x7f4ad8101fe0 2024-04-23T18:19:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.718+0000 7f4ac77fe640 1 -- 172.21.15.32:0/2025659276 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4acc031b50 con 0x7f4ad8101fe0 2024-04-23T18:19:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.718+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad81aba20 con 0x7f4ad8101fe0 2024-04-23T18:19:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.718+0000 7f4ac77fe640 1 -- 172.21.15.32:0/2025659276 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc035030 con 0x7f4ad8101fe0 2024-04-23T18:19:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.721+0000 7f4adf768740 1 Processor -- start 2024-04-23T18:19:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 -- start start 2024-04-23T18:19:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ac77fe640 1 -- 172.21.15.32:0/2025659276 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4acc042070 con 0x7f4ad8101fe0 2024-04-23T18:19:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8babc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5574ba81f970 con 0x5574ba8c59c0 2024-04-23T18:19:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5574ba7bc4a0 con 0x5574ba8bb100 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4adf768740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574ba81b930 con 0x5574ba8ba7e0 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ad659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ad5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ad5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45574/0 (socket says 172.21.15.32:45574) 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/2384675612 learned_addr learned my addr 172.21.15.32:0/2384675612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.722+0000 7f4ad6d9d640 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8babc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/2384675612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 msgr2=0x5574ba8babc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8babc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/2384675612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 msgr2=0x5574ba8c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/2384675612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ba532210 con 0x5574ba8c59c0 2024-04-23T18:19:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ac77fe640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 0x7f4ac00a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009c20 tx=0x7f4ac8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3dcace86a7b042 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.723+0000 7f4ac77fe640 1 -- 172.21.15.32:0/2025659276 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4acc03b9a0 con 0x7f4ad8101fe0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.724+0000 7f4ac4ff9640 1 -- 172.21.15.32:0/2384675612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8004050 con 0x5574ba8c59c0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.724+0000 7f4ac4ff9640 1 -- 172.21.15.32:0/2384675612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac80054d0 con 0x5574ba8c59c0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4ad6d9d640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 0x7f4ac00a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4ac4ff9640 1 -- 172.21.15.32:0/2384675612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8004d00 con 0x5574ba8c59c0 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 -- 172.21.15.32:0/2384675612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 msgr2=0x5574ba8c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009c20 tx=0x7f4ac8011810 comp rx=0 tx=0).stop 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 -- 172.21.15.32:0/2384675612 shutdown_connections 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c7db0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 --2- 172.21.15.32:0/2384675612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8babc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4adf768740 1 -- 172.21.15.32:0/2384675612 >> 172.21.15.32:0/2384675612 conn(0x5574ba8b6cf0 msgr2=0x5574ba8b70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.725+0000 7f4ad6d9d640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 0x7f4ac00a6430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0007e50 tx=0x7f4ad0007520 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 -- 172.21.15.32:0/2384675612 shutdown_connections 2024-04-23T18:19:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 -- 172.21.15.32:0/2384675612 wait complete. 2024-04-23T18:19:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 -- start start 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 Processor -- start 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.726+0000 7f4adf768740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4adf768740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4adf768740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5574ba8c85f0 con 0x5574ba8c59c0 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4adf768740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5574ba7ba0b0 con 0x5574ba8bb100 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4adf768740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574ba8c8770 con 0x5574ba8ba7e0 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4adf768740 10 client.0 _ll_register_callbacks cb 0x5574ba7a7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:51102/0 (socket says 172.21.15.32:51102) 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/4013823208 learned_addr learned my addr 172.21.15.32:0/4013823208 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad659c640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/4013823208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 msgr2=0x5574ba8c52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/4013823208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 msgr2=0x5574ba8c2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.727+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/4013823208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ba5fe620 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.728+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ac801b040 tx=0x7f4ac8004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=545286581a9b3a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.728+0000 7f4abdffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.728+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8029070 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.728+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ac80054d0 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.728+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4ac8003a00 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ac8035050 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 10 client.25652 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574ba8d45b0 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac80054d0 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f4ac8005160 con 0x5574ba8bb100 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 10 client.25652 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 10 client.25652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 20 client.25652 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba8d5e00 0x5574ba8d8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 4 client.25652 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574ba82d390 con 0x5574ba8d5e00 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.729+0000 7f4ad659c640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba8d5e00 0x5574ba8d8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.730+0000 7f4abdffb640 10 client.25652 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.730+0000 7f4ad659c640 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba8d5e00 0x5574ba8d8260 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4678f89fc632cb7d server_cookie=cfbbabad151c8c94 in_seq=0 out_seq=0 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.731+0000 7f4abdffb640 1 -- 172.21.15.32:0/4013823208 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574ba82d390 con 0x5574ba8d5e00 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.731+0000 7f4abdffb640 10 client.25652 handle_command_reply: tid=0 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.731+0000 7f4adf768740 1 client.25652 shutdown 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 msgr2=0x5574ba8c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ac801b040 tx=0x7f4ac8004740 comp rx=0 tx=0).stop 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 shutdown_connections 2024-04-23T18:19:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba8d5e00 0x5574ba8d8260 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8c2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8bb100 0x5574ba8c25b0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ac801b040 tx=0x7f4ac8004740 comp rx=0 tx=0).stop 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 --2- 172.21.15.32:0/4013823208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 >> 172.21.15.32:0/4013823208 conn(0x5574ba8b6cf0 msgr2=0x5574ba8bca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4abdffb640 0 client.25652 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.732+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 shutdown_connections 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.733+0000 7f4adf768740 1 -- 172.21.15.32:0/4013823208 wait complete. 2024-04-23T18:19:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.733+0000 7f4adf768740 20 client.25652 trim_cache size 0 max 16384 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.763+0000 7f4adf768740 1 Processor -- start 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.763+0000 7f4adf768740 1 -- start start 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 0x5574ba8de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab8010ff0 con 0x5574ba8c59c0 2024-04-23T18:19:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab800e890 con 0x5574ba8c5330 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4adf768740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab800c6e0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4ad6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 0x5574ba8de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4ad6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 0x5574ba8de0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:51104/0 (socket says 172.21.15.32:51104) 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.764+0000 7f4ad6d9d640 1 -- 172.21.15.32:0/1837530341 learned_addr learned my addr 172.21.15.32:0/1837530341 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad659c640 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/1837530341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 msgr2=0x5574ba8c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/1837530341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 msgr2=0x5574ba8de0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 0x5574ba8de0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.766+0000 7f4ad5d9b640 1 -- 172.21.15.32:0/1837530341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ba8baef0 con 0x5574ba8c59c0 2024-04-23T18:19:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.767+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8ddb70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x5574ba8c3640 tx=0x7f4ac802e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7742bacf9f0f060 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.767+0000 7f4abffff640 1 -- 172.21.15.32:0/1837530341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8029070 con 0x5574ba8c59c0 2024-04-23T18:19:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.767+0000 7f4abffff640 1 -- 172.21.15.32:0/1837530341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac80097a0 con 0x5574ba8c59c0 2024-04-23T18:19:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.768+0000 7f4abffff640 1 -- 172.21.15.32:0/1837530341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8029070 con 0x5574ba8c59c0 2024-04-23T18:19:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.769+0000 7f4adf768740 1 -- 172.21.15.32:0/1837530341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 msgr2=0x5574ba8ddb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.769+0000 7f4adf768740 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8ddb70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x5574ba8c3640 tx=0x7f4ac802e960 comp rx=0 tx=0).stop 2024-04-23T18:19:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.770+0000 7f4adf768740 1 -- 172.21.15.32:0/1837530341 shutdown_connections 2024-04-23T18:19:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.770+0000 7f4adf768740 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c5330 0x5574ba8de0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.770+0000 7f4adf768740 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c59c0 0x5574ba8ddb70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.770+0000 7f4adf768740 1 --2- 172.21.15.32:0/1837530341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.770+0000 7f4adf768740 1 -- 172.21.15.32:0/1837530341 >> 172.21.15.32:0/1837530341 conn(0x5574ba8b6cf0 msgr2=0x5574ba8c2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.771+0000 7f4adf768740 1 -- 172.21.15.32:0/1837530341 shutdown_connections 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 -- 172.21.15.32:0/1837530341 wait complete. 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 -- start start 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 Processor -- start 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c5330 0x5574ba8cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.773+0000 7f4adf768740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c59c0 0x5574ba8c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4adf768740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab800cd50 con 0x5574ba8c5330 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4adf768740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab800e1c0 con 0x5574ba8c59c0 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4adf768740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab800d420 con 0x5574ba8ba7e0 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4adf768740 10 client.0 _ll_register_callbacks cb 0x7f4ac80043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4ad659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4ad659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36828/0 (socket says 172.21.15.32:36828) 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.774+0000 7f4ad659c640 1 -- 172.21.15.32:0/3862036862 learned_addr learned my addr 172.21.15.32:0/3862036862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.775+0000 7f4ad659c640 1 -- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c59c0 msgr2=0x5574ba8c9000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:19:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.775+0000 7f4ad659c640 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c59c0 0x5574ba8c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.775+0000 7f4ad659c640 1 -- 172.21.15.32:0/3862036862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c5330 msgr2=0x5574ba8cdfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:19:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.775+0000 7f4ad659c640 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c5330 0x5574ba8cdfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.775+0000 7f4ad659c640 1 -- 172.21.15.32:0/3862036862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ba8c97d0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.776+0000 7f4ad659c640 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4acc005ab0 tx=0x7f4acc009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90e296dbd4b48fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.776+0000 7f4abdffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.776+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0d5070 con 0x5574ba8ba7e0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.776+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4acc0b95f0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.776+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ac80097a0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.777+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4acc0bcf80 con 0x5574ba8ba7e0 2024-04-23T18:19:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.778+0000 7f4adf768740 10 client.25639 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.778+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574ba94d0c0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac80097a0 con 0x5574ba8ba7e0 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f4acc03b460 con 0x5574ba8ba7e0 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 10 client.25639 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 20 client.25639 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba94d7f0 0x5574ba94fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 4 client.25639 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574ba827200 con 0x5574ba94d7f0 2024-04-23T18:19:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.781+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba94d7f0 0x5574ba94fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.783+0000 7f4ad5d9b640 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba94d7f0 0x5574ba94fc50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6d7af8e3931736f server_cookie=46a4f97a575ada53 in_seq=0 out_seq=0 2024-04-23T18:19:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.785+0000 7f4abdffb640 10 client.25639 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4abdffb640 1 -- 172.21.15.32:0/3862036862 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574ba827200 con 0x5574ba94d7f0 2024-04-23T18:19:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4abdffb640 10 client.25639 handle_command_reply: tid=0 2024-04-23T18:19:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 client.25639 shutdown 2024-04-23T18:19:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 msgr2=0x5574ba8cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4acc005ab0 tx=0x7f4acc009680 comp rx=0 tx=0).stop 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 shutdown_connections 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5574ba94d7f0 0x5574ba94fc50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574ba8c59c0 0x5574ba8c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5574ba8c5330 0x5574ba8cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4abdffb640 0 client.25639 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 --2- 172.21.15.32:0/3862036862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574ba8ba7e0 0x5574ba8cda90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 >> 172.21.15.32:0/3862036862 conn(0x5574ba8b6cf0 msgr2=0x5574ba8bc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.799+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 shutdown_connections 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4adf768740 1 -- 172.21.15.32:0/3862036862 wait complete. 2024-04-23T18:19:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4adf768740 20 client.25639 trim_cache size 0 max 16384 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 msgr2=0x7f4ac00a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 0x7f4ac00a6430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0007e50 tx=0x7f4ad0007520 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 msgr2=0x7f4ad81a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4acc031440 tx=0x7f4acc003c00 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 shutdown_connections 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4ac00a3f70 0x7f4ac00a6430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad810ddd0 0x7f4ad810e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad81070b0 0x7f4ad81a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 --2- 172.21.15.32:0/2025659276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ad8101fe0 0x7f4ad81a48d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 >> 172.21.15.32:0/2025659276 conn(0x7f4ad80fd900 msgr2=0x7f4ad80ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.800+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 shutdown_connections 2024-04-23T18:19:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:21.801+0000 7f4ad759e640 1 -- 172.21.15.32:0/2025659276 wait complete. 2024-04-23T18:19:22.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:22 smithi040 ceph-mon[32626]: pgmap v131: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-23T18:19:22.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:22 smithi064 ceph-mon[33062]: pgmap v131: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-23T18:19:23.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:22 smithi032 ceph-mon[29221]: pgmap v131: 97 pgs: 97 active+clean; 2.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-23T18:19:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 235.0 s, 3148.8 tps, lat 31.735 ms stddev 33.306 2024-04-23T18:19:24.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:24 smithi040 ceph-mon[32626]: pgmap v132: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-04-23T18:19:24.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:24 smithi064 ceph-mon[33062]: pgmap v132: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-04-23T18:19:25.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:24 smithi032 ceph-mon[29221]: pgmap v132: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-04-23T18:19:26.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:26 smithi040 ceph-mon[32626]: pgmap v133: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:26.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:26 smithi064 ceph-mon[33062]: pgmap v133: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:26 smithi032 ceph-mon[29221]: pgmap v133: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 240.0 s, 3335.0 tps, lat 30.012 ms stddev 33.786 2024-04-23T18:19:28.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:28 smithi040 ceph-mon[32626]: pgmap v134: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:28.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:28 smithi064 ceph-mon[33062]: pgmap v134: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:29.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:28 smithi032 ceph-mon[29221]: pgmap v134: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.6 MiB/s wr, 624 op/s 2024-04-23T18:19:30.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:30 smithi040 ceph-mon[32626]: pgmap v135: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 935 op/s 2024-04-23T18:19:30.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:30 smithi064 ceph-mon[33062]: pgmap v135: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 935 op/s 2024-04-23T18:19:31.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:30 smithi032 ceph-mon[29221]: pgmap v135: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 935 op/s 2024-04-23T18:19:32.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:32 smithi040 ceph-mon[32626]: pgmap v136: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 613 op/s 2024-04-23T18:19:32.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:32 smithi064 ceph-mon[33062]: pgmap v136: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 613 op/s 2024-04-23T18:19:33.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:32 smithi032 ceph-mon[29221]: pgmap v136: 97 pgs: 97 active+clean; 2.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 613 op/s 2024-04-23T18:19:33.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 245.0 s, 3041.6 tps, lat 32.773 ms stddev 37.151 2024-04-23T18:19:34.866 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:34 smithi040 ceph-mon[32626]: pgmap v137: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 879 op/s 2024-04-23T18:19:34.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:34 smithi064 ceph-mon[33062]: pgmap v137: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 879 op/s 2024-04-23T18:19:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.944+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2929224869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8811b730 msgr2=0x7f1a8811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.944+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2929224869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8811b730 0x7f1a8811db20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009be0 tx=0x7f1a8002a710 comp rx=0 tx=0).stop 2024-04-23T18:19:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.946+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2929224869 shutdown_connections 2024-04-23T18:19:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.946+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2929224869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8811b730 0x7f1a8811db20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.946+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2929224869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8810e7d0 0x7f1a8810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.946+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2929224869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a881106f0 0x7f1a8810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.946+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2929224869 >> 172.21.15.32:0/2929224869 conn(0x7f1a8806d030 msgr2=0x7f1a8806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.949+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2929224869 shutdown_connections 2024-04-23T18:19:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.949+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2929224869 wait complete. 2024-04-23T18:19:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.950+0000 7f1a8fc7c640 1 Processor -- start 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 -- start start 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a881106f0 0x7f1a881a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8811b730 0x7f1a881abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88120f90 con 0x7f1a881106f0 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88120e10 con 0x7f1a8811b730 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8fc7c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88121110 con 0x7f1a8810e7d0 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8da18640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8da18640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53682/0 (socket says 172.21.15.32:53682) 2024-04-23T18:19:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8da18640 1 -- 172.21.15.32:0/2911372945 learned_addr learned my addr 172.21.15.32:0/2911372945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.951+0000 7f1a8d217640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a881106f0 0x7f1a881a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8e219640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8811b730 0x7f1a881abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 -- 172.21.15.32:0/2911372945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8811b730 msgr2=0x7f1a881abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8811b730 0x7f1a881abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 -- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a881106f0 msgr2=0x7f1a881a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a881106f0 0x7f1a881a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 -- 172.21.15.32:0/2911372945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a881ac4e0 con 0x7f1a8810e7d0 2024-04-23T18:19:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.952+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c890 tx=0x7f1a8400cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.956+0000 7f1a7effd640 1 -- 172.21.15.32:0/2911372945 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004220 con 0x7f1a8810e7d0 2024-04-23T18:19:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.956+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a881ac7d0 con 0x7f1a8810e7d0 2024-04-23T18:19:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.956+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a881b4060 con 0x7f1a8810e7d0 2024-04-23T18:19:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.956+0000 7f1a7effd640 1 -- 172.21.15.32:0/2911372945 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a840043c0 con 0x7f1a8810e7d0 2024-04-23T18:19:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.956+0000 7f1a7effd640 1 -- 172.21.15.32:0/2911372945 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004a00 con 0x7f1a8810e7d0 2024-04-23T18:19:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.960+0000 7f1a7effd640 1 -- 172.21.15.32:0/2911372945 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1a84004ba0 con 0x7f1a8810e7d0 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.961+0000 7f1a92997740 1 Processor -- start 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.961+0000 7f1a92997740 1 -- start start 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84b1800 0x5604f84b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5604f840b970 con 0x5604f84a6620 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604f83a84a0 con 0x5604f84b1800 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a92997740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5604f8407930 con 0x5604f84a6f40 2024-04-23T18:19:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a7effd640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 0x7f1a740a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a8e219640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a8e219640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57510/0 (socket says 172.21.15.32:57510) 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.962+0000 7f1a8e219640 1 -- 172.21.15.32:0/925630252 learned_addr learned my addr 172.21.15.32:0/925630252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a7effd640 1 -- 172.21.15.32:0/2911372945 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1a840d1710 con 0x7f1a8810e7d0 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 0x7f1a740a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8da18640 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 -- 172.21.15.32:0/925630252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 msgr2=0x5604f84b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 -- 172.21.15.32:0/925630252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84b1800 msgr2=0x5604f84b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84b1800 0x5604f84b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 -- 172.21.15.32:0/925630252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604f811e210 con 0x5604f84a6620 2024-04-23T18:19:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.963+0000 7f1a8e219640 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1a80031040 tx=0x7f1a80030980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509b22272f532d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.964+0000 7f1a73fff640 1 -- 172.21.15.32:0/925630252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8004d070 con 0x5604f84a6620 2024-04-23T18:19:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.964+0000 7f1a8e219640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 0x7f1a740a63e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a8806b5d0 tx=0x7f1a80045000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.964+0000 7f1a73fff640 1 -- 172.21.15.32:0/925630252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a80048bb0 con 0x5604f84a6620 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.964+0000 7f1a73fff640 1 -- 172.21.15.32:0/925630252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80036040 con 0x5604f84a6620 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- 172.21.15.32:0/925630252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 msgr2=0x5604f84a6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1a80031040 tx=0x7f1a80030980 comp rx=0 tx=0).stop 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- 172.21.15.32:0/925630252 shutdown_connections 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84b1800 0x5604f84b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 --2- 172.21.15.32:0/925630252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84a6620 0x5604f84a6a00 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- 172.21.15.32:0/925630252 >> 172.21.15.32:0/925630252 conn(0x5604f84a2b30 msgr2=0x5604f84a2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- 172.21.15.32:0/925630252 shutdown_connections 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- 172.21.15.32:0/925630252 wait complete. 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.965+0000 7f1a92997740 1 -- start start 2024-04-23T18:19:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 Processor -- start 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5604f84b43d0 con 0x5604f84b1800 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604f83a60b0 con 0x5604f84a6f40 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5604f84b4550 con 0x5604f84a6620 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a8d217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a8d217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46858/0 (socket says 172.21.15.32:46858) 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a8d217640 1 -- 172.21.15.32:0/3283746964 learned_addr learned my addr 172.21.15.32:0/3283746964 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a92997740 10 client.0 _ll_register_callbacks cb 0x5604f8393d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.966+0000 7f1a8e219640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8da18640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8d217640 1 -- 172.21.15.32:0/3283746964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 msgr2=0x5604f84b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8d217640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8d217640 1 -- 172.21.15.32:0/3283746964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 msgr2=0x5604f84ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8d217640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.967+0000 7f1a8d217640 1 -- 172.21.15.32:0/3283746964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604f81ea620 con 0x5604f84a6f40 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.968+0000 7f1a8d217640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a78005fd0 tx=0x7f1a78011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d0630844ecc3b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.968+0000 7f1a717fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7801e060 con 0x5604f84a6f40 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1a7800fcb0 con 0x5604f84a6f40 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a80048bb0 con 0x5604f84a6f40 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a78023460 con 0x5604f84a6f40 2024-04-23T18:19:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a92997740 10 client.25661 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604f84c01a0 con 0x5604f84a6f40 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.969+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a80048bb0 con 0x5604f84a6f40 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f1a7800e440 con 0x5604f84a6f40 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 10 client.25661 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 20 client.25661 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f84c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 4 client.25661 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604f8419390 con 0x5604f84c1660 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.970+0000 7f1a8da18640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f84c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.971+0000 7f1a717fa640 10 client.25661 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.971+0000 7f1a8da18640 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f84c3a50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0532d0b081a683c server_cookie=e47d69ea39771826 in_seq=0 out_seq=0 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.971+0000 7f1a717fa640 1 -- 172.21.15.32:0/3283746964 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604f8419390 con 0x5604f84c1660 2024-04-23T18:19:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.971+0000 7f1a717fa640 10 client.25661 handle_command_reply: tid=0 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.971+0000 7f1a92997740 1 client.25661 shutdown 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 msgr2=0x5604f84ae450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a78005fd0 tx=0x7f1a78011b50 comp rx=0 tx=0).stop 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 shutdown_connections 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f84c3a50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a717fa640 0 client.25661 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84ae450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 --2- 172.21.15.32:0/3283746964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 >> 172.21.15.32:0/3283746964 conn(0x5604f84a2b30 msgr2=0x5604f84a8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 shutdown_connections 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 1 -- 172.21.15.32:0/3283746964 wait complete. 2024-04-23T18:19:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.972+0000 7f1a92997740 20 client.25661 trim_cache size 0 max 16384 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 Processor -- start 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 -- start start 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c010f40 con 0x5604f84b1800 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c00e7e0 con 0x5604f84a6620 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a92997740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c00c630 con 0x5604f84a6f40 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a8da18640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a8da18640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46870/0 (socket says 172.21.15.32:46870) 2024-04-23T18:19:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.993+0000 7f1a8da18640 1 -- 172.21.15.32:0/2494748224 learned_addr learned my addr 172.21.15.32:0/2494748224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8e219640 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8d217640 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 -- 172.21.15.32:0/2494748224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 msgr2=0x5604f84b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 -- 172.21.15.32:0/2494748224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 msgr2=0x5604f84c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 -- 172.21.15.32:0/2494748224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604f84c3dc0 con 0x5604f84a6620 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a8da18640 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a84005e50 tx=0x7f1a840d1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3aa0b42732578e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a737fe640 1 -- 172.21.15.32:0/2494748224 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a840bb070 con 0x5604f84a6620 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a737fe640 1 -- 172.21.15.32:0/2494748224 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a8409b9a0 con 0x5604f84a6620 2024-04-23T18:19:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.994+0000 7f1a737fe640 1 -- 172.21.15.32:0/2494748224 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8401d070 con 0x5604f84a6620 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.995+0000 7f1a92997740 1 -- 172.21.15.32:0/2494748224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 msgr2=0x5604f84bd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.995+0000 7f1a92997740 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a84005e50 tx=0x7f1a840d1040 comp rx=0 tx=0).stop 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 -- 172.21.15.32:0/2494748224 shutdown_connections 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f84c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6f40 0x5604f84b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 --2- 172.21.15.32:0/2494748224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6620 0x5604f84bd2c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 -- 172.21.15.32:0/2494748224 >> 172.21.15.32:0/2494748224 conn(0x5604f84a2b30 msgr2=0x5604f84a99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.996+0000 7f1a92997740 1 -- 172.21.15.32:0/2494748224 shutdown_connections 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.997+0000 7f1a92997740 1 -- 172.21.15.32:0/2494748224 wait complete. 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.997+0000 7f1a92997740 1 -- start start 2024-04-23T18:19:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.997+0000 7f1a92997740 1 Processor -- start 2024-04-23T18:19:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f8526bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c00cca0 con 0x5604f84b1800 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c00e110 con 0x5604f84a6f40 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c00d370 con 0x5604f84a6620 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8da18640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46876/0 (socket says 172.21.15.32:46876) 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 -- 172.21.15.32:0/599896091 learned_addr learned my addr 172.21.15.32:0/599896091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8e219640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f8526bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a92997740 10 client.0 _ll_register_callbacks cb 0x7f1a78009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 -- 172.21.15.32:0/599896091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 msgr2=0x5604f84ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 -- 172.21.15.32:0/599896091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 msgr2=0x5604f8526bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f8526bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8d217640 1 -- 172.21.15.32:0/599896091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604f84ad460 con 0x5604f84a6f40 2024-04-23T18:19:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8e219640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f8526bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.998+0000 7f1a8da18640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a8d217640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009d90 tx=0x7f1a7800fce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee35c5e05fd0dae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a717fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78032060 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1a7800ec90 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a8409b9a0 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a7800ee90 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a92997740 10 client.25667 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604f8528210 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:34.999+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a780054f0 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f1a78004bd0 con 0x5604f84a6f40 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 10 client.25667 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 10 client.25667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 20 client.25667 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f852aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 4 client.25667 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f1a8409b9a0 con 0x5604f84c1660 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.000+0000 7f1a8da18640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f852aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.001+0000 7f1a717fa640 10 client.25667 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.001+0000 7f1a8da18640 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f852aa00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5662ab377659dc5 server_cookie=e856254abdffa77d in_seq=0 out_seq=0 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.012+0000 7f1a717fa640 1 -- 172.21.15.32:0/599896091 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f1a8409b9a0 con 0x5604f84c1660 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.012+0000 7f1a717fa640 10 client.25667 handle_command_reply: tid=0 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.012+0000 7f1a92997740 1 client.25667 shutdown 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 msgr2=0x5604f84abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009d90 tx=0x7f1a7800fce0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 shutdown_connections 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604f84c1660 0x5604f852aa00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a717fa640 0 client.25667 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604f84b1800 0x5604f8526bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f84a6f40 0x5604f84abdb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 --2- 172.21.15.32:0/599896091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f84a6620 0x5604f84ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 >> 172.21.15.32:0/599896091 conn(0x5604f84a2b30 msgr2=0x5604f84a7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 shutdown_connections 2024-04-23T18:19:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 1 -- 172.21.15.32:0/599896091 wait complete. 2024-04-23T18:19:35.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:34 smithi032 ceph-mon[29221]: pgmap v137: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 879 op/s 2024-04-23T18:19:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.013+0000 7f1a92997740 20 client.25667 trim_cache size 0 max 16384 2024-04-23T18:19:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:19:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:19:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:19:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.015+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 msgr2=0x7f1a740a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.015+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 0x7f1a740a63e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a8806b5d0 tx=0x7f1a80045000 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.015+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 msgr2=0x7f1a881a74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.015+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c890 tx=0x7f1a8400cd50 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 shutdown_connections 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8811b730 0x7f1a881abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a740a3f20 0x7f1a740a63e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a881106f0 0x7f1a881a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 --2- 172.21.15.32:0/2911372945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a8810e7d0 0x7f1a881a74f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 >> 172.21.15.32:0/2911372945 conn(0x7f1a8806d030 msgr2=0x7f1a8811a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 shutdown_connections 2024-04-23T18:19:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:35.016+0000 7f1a8fc7c640 1 -- 172.21.15.32:0/2911372945 wait complete. 2024-04-23T18:19:36.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:36 smithi064 ceph-mon[33062]: pgmap v138: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:36 smithi032 ceph-mon[29221]: pgmap v138: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:36 smithi040 ceph-mon[32626]: pgmap v138: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 250.0 s, 3423.6 tps, lat 29.312 ms stddev 32.953 2024-04-23T18:19:38.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:38 smithi064 ceph-mon[33062]: pgmap v139: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:39.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:38 smithi032 ceph-mon[29221]: pgmap v139: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:39.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:38 smithi040 ceph-mon[32626]: pgmap v139: 97 pgs: 97 active+clean; 2.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 609 op/s 2024-04-23T18:19:40.917 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:40 smithi040 ceph-mon[32626]: pgmap v140: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 990 op/s 2024-04-23T18:19:40.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:40 smithi064 ceph-mon[33062]: pgmap v140: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 990 op/s 2024-04-23T18:19:41.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:40 smithi032 ceph-mon[29221]: pgmap v140: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 990 op/s 2024-04-23T18:19:42.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:42 smithi064 ceph-mon[33062]: pgmap v141: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.8 MiB/s wr, 679 op/s 2024-04-23T18:19:43.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:42 smithi032 ceph-mon[29221]: pgmap v141: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.8 MiB/s wr, 679 op/s 2024-04-23T18:19:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:42 smithi040 ceph-mon[32626]: pgmap v141: 97 pgs: 97 active+clean; 2.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.8 MiB/s wr, 679 op/s 2024-04-23T18:19:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 255.0 s, 3323.4 tps, lat 29.812 ms stddev 31.301 2024-04-23T18:19:44.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:44 smithi064 ceph-mon[33062]: pgmap v142: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-23T18:19:45.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:44 smithi032 ceph-mon[29221]: pgmap v142: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-23T18:19:45.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:44 smithi040 ceph-mon[32626]: pgmap v142: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-23T18:19:46.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:46 smithi064 ceph-mon[33062]: pgmap v143: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 722 op/s 2024-04-23T18:19:47.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:46 smithi032 ceph-mon[29221]: pgmap v143: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 722 op/s 2024-04-23T18:19:47.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:46 smithi040 ceph-mon[32626]: pgmap v143: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 722 op/s 2024-04-23T18:19:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.157+0000 7f923af4b640 1 -- 172.21.15.32:0/2431598178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076480 msgr2=0x7f9234076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.157+0000 7f923af4b640 1 --2- 172.21.15.32:0/2431598178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076480 0x7f9234076860 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9224009ca0 tx=0x7f922402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.158+0000 7f923af4b640 1 -- 172.21.15.32:0/2431598178 shutdown_connections 2024-04-23T18:19:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.158+0000 7f923af4b640 1 --2- 172.21.15.32:0/2431598178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f9234114ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.158+0000 7f923af4b640 1 --2- 172.21.15.32:0/2431598178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076da0 0x7f92341126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.158+0000 7f923af4b640 1 --2- 172.21.15.32:0/2431598178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076480 0x7f9234076860 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.158+0000 7f923af4b640 1 -- 172.21.15.32:0/2431598178 >> 172.21.15.32:0/2431598178 conn(0x7f92340ffa30 msgr2=0x7f9234101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.159+0000 7f923af4b640 1 -- 172.21.15.32:0/2431598178 shutdown_connections 2024-04-23T18:19:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.159+0000 7f923af4b640 1 -- 172.21.15.32:0/2431598178 wait complete. 2024-04-23T18:19:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 Processor -- start 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 -- start start 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 0x7f92341a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076da0 0x7f92341a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f923419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9234118670 con 0x7f9234076da0 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92341184f0 con 0x7f9234076480 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923af4b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f92341187f0 con 0x7f9234112c00 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f9239f49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 0x7f92341a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f9239f49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 0x7f92341a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45802/0 (socket says 172.21.15.32:45802) 2024-04-23T18:19:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f9239f49640 1 -- 172.21.15.32:0/2276662200 learned_addr learned my addr 172.21.15.32:0/2276662200 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.160+0000 7f923a74a640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f923419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f9239748640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076da0 0x7f92341a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 -- 172.21.15.32:0/2276662200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 msgr2=0x7f92341a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 0x7f92341a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 -- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076da0 msgr2=0x7f92341a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076da0 0x7f92341a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 -- 172.21.15.32:0/2276662200 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923419f060 con 0x7f9234112c00 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f923a74a640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f923419e890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f923406b340 tx=0x7f922c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.161+0000 7f9232ffd640 1 -- 172.21.15.32:0/2276662200 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c004220 con 0x7f9234112c00 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.162+0000 7f9232ffd640 1 -- 172.21.15.32:0/2276662200 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f922c0043c0 con 0x7f9234112c00 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.162+0000 7f9232ffd640 1 -- 172.21.15.32:0/2276662200 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c003de0 con 0x7f9234112c00 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.162+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923419f350 con 0x7f9234112c00 2024-04-23T18:19:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.162+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92341ab520 con 0x7f9234112c00 2024-04-23T18:19:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.165+0000 7f923ef9d740 1 Processor -- start 2024-04-23T18:19:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f9232ffd640 1 -- 172.21.15.32:0/2276662200 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f922c00f450 con 0x7f9234112c00 2024-04-23T18:19:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 -- start start 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2d7e0 0x565512d2dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d389c0 0x565512d3adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565512c92970 con 0x565512d2d7e0 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565512c2f4a0 con 0x565512d2e100 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.166+0000 7f923ef9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x565512c8e930 con 0x565512d389c0 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45818/0 (socket says 172.21.15.32:45818) 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 -- 172.21.15.32:0/2947541884 learned_addr learned my addr 172.21.15.32:0/2947541884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239748640 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d389c0 0x565512d3adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239748640 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2d7e0 0x565512d2dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 -- 172.21.15.32:0/2947541884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d389c0 msgr2=0x565512d3adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d389c0 0x565512d3adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 -- 172.21.15.32:0/2947541884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2d7e0 msgr2=0x565512d2dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2d7e0 0x565512d2dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 -- 172.21.15.32:0/2947541884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655129a5210 con 0x565512d2e100 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9239f49640 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f922402f3f0 tx=0x7f922402f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64d92bb64862b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f9232ffd640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 0x7f921c0a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.167+0000 7f921bfff640 1 -- 172.21.15.32:0/2947541884 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224031650 con 0x565512d2e100 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.168+0000 7f9239748640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 0x7f921c0a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.168+0000 7f9232ffd640 1 -- 172.21.15.32:0/2276662200 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f922c0cfff0 con 0x7f9234112c00 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.168+0000 7f921bfff640 1 -- 172.21.15.32:0/2947541884 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9224036030 con 0x565512d2e100 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.168+0000 7f921bfff640 1 -- 172.21.15.32:0/2947541884 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224004460 con 0x565512d2e100 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 -- 172.21.15.32:0/2947541884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 msgr2=0x565512d38480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f9239748640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 0x7f921c0a6220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x565512d2b110 tx=0x7f9228004730 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f922402f3f0 tx=0x7f922402f420 comp rx=0 tx=0).stop 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 -- 172.21.15.32:0/2947541884 shutdown_connections 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d389c0 0x565512d3adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2e100 0x565512d38480 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2947541884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2d7e0 0x565512d2dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 -- 172.21.15.32:0/2947541884 >> 172.21.15.32:0/2947541884 conn(0x565512d29cf0 msgr2=0x565512d2a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.169+0000 7f923ef9d740 1 -- 172.21.15.32:0/2947541884 shutdown_connections 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 -- 172.21.15.32:0/2947541884 wait complete. 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 -- start start 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 Processor -- start 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d35570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d35ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.170+0000 7f923ef9d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565512d3b410 con 0x565512d2e100 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f923ef9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565512c2d0b0 con 0x565512d389c0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f923ef9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x565512d3b590 con 0x565512d2d7e0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f923ef9d740 10 client.0 _ll_register_callbacks cb 0x565512c1ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239748640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d35570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:43572/0 (socket says 172.21.15.32:43572) 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239748640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d35570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41518/0 (socket says 172.21.15.32:41518) 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 -- 172.21.15.32:0/783188891 learned_addr learned my addr 172.21.15.32:0/783188891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f923a74a640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d35ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 -- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 msgr2=0x565512d35ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d35ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 -- 172.21.15.32:0/783188891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 msgr2=0x565512d35570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d35570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 -- 172.21.15.32:0/783188891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565512a71620 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.171+0000 7f9239f49640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f922402aeb0 tx=0x7f9224003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be07297c984fc53f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f92197fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922404a040 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f92240041c0 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9224036030 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f92240098c0 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f923ef9d740 10 client.25663 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.172+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x565512d3b410 con 0x565512d2d7e0 2024-04-23T18:19:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9224036030 con 0x565512d2d7e0 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f922404d030 con 0x565512d2d7e0 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 10 client.25663 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 20 client.25663 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512d48490 0x565512d4a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 4 client.25663 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565512ca0390 con 0x565512d48490 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.173+0000 7f9239748640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512d48490 0x565512d4a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.174+0000 7f9239748640 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512d48490 0x565512d4a880 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52b07a7235ad5804 server_cookie=6dd561b0f0ad8912 in_seq=0 out_seq=0 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.174+0000 7f92197fa640 10 client.25663 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.174+0000 7f92197fa640 1 -- 172.21.15.32:0/783188891 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565512ca0390 con 0x565512d48490 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.174+0000 7f92197fa640 10 client.25663 handle_command_reply: tid=0 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.174+0000 7f923ef9d740 1 client.25663 shutdown 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 msgr2=0x565512d38280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f922402aeb0 tx=0x7f9224003ba0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 shutdown_connections 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512d48490 0x565512d4a880 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d35ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d35570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 --2- 172.21.15.32:0/783188891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d38280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f92197fa640 0 client.25663 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 >> 172.21.15.32:0/783188891 conn(0x565512d29cf0 msgr2=0x565512d2fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 shutdown_connections 2024-04-23T18:19:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 1 -- 172.21.15.32:0/783188891 wait complete. 2024-04-23T18:19:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.176+0000 7f923ef9d740 20 client.25663 trim_cache size 0 max 16384 2024-04-23T18:19:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.200+0000 7f923ef9d740 1 Processor -- start 2024-04-23T18:19:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.200+0000 7f923ef9d740 1 -- start start 2024-04-23T18:19:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2d7e0 0x565512d44300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2e100 0x565512d30300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b6670 con 0x565512d389c0 2024-04-23T18:19:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b3f10 con 0x565512d2d7e0 2024-04-23T18:19:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923ef9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b1d60 con 0x565512d2e100 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41532/0 (socket says 172.21.15.32:41532) 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 -- 172.21.15.32:0/1123515907 learned_addr learned my addr 172.21.15.32:0/1123515907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f9239748640 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2e100 0x565512d30300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f9239f49640 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2d7e0 0x565512d44300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 -- 172.21.15.32:0/1123515907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2e100 msgr2=0x565512d30300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2e100 0x565512d30300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 -- 172.21.15.32:0/1123515907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2d7e0 msgr2=0x565512d44300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2d7e0 0x565512d44300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.201+0000 7f923a74a640 1 -- 172.21.15.32:0/1123515907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565512d256b0 con 0x565512d389c0 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.202+0000 7f923a74a640 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f922c09bf10 tx=0x7f922c09be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4470b9cef7abefa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.202+0000 7f921b7fe640 1 -- 172.21.15.32:0/1123515907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0b4020 con 0x565512d389c0 2024-04-23T18:19:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.202+0000 7f921b7fe640 1 -- 172.21.15.32:0/1123515907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f922c0a0330 con 0x565512d389c0 2024-04-23T18:19:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.202+0000 7f921b7fe640 1 -- 172.21.15.32:0/1123515907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c01d040 con 0x565512d389c0 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 -- 172.21.15.32:0/1123515907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 msgr2=0x565512d37db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f922c09bf10 tx=0x7f922c09be80 comp rx=0 tx=0).stop 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 -- 172.21.15.32:0/1123515907 shutdown_connections 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d389c0 0x565512d37db0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2e100 0x565512d30300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 --2- 172.21.15.32:0/1123515907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d2d7e0 0x565512d44300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.203+0000 7f923ef9d740 1 -- 172.21.15.32:0/1123515907 >> 172.21.15.32:0/1123515907 conn(0x565512d29cf0 msgr2=0x565512d28de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.204+0000 7f923ef9d740 1 -- 172.21.15.32:0/1123515907 shutdown_connections 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.204+0000 7f923ef9d740 1 -- 172.21.15.32:0/1123515907 wait complete. 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.204+0000 7f923ef9d740 1 -- start start 2024-04-23T18:19:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.204+0000 7f923ef9d740 1 Processor -- start 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d3c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d3c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b23d0 con 0x565512d2e100 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b3840 con 0x565512d389c0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f92341b2aa0 con 0x565512d2d7e0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923ef9d740 10 client.0 _ll_register_callbacks cb 0x7f9224031810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923a74a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923a74a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45848/0 (socket says 172.21.15.32:45848) 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f923a74a640 1 -- 172.21.15.32:0/2886060758 learned_addr learned my addr 172.21.15.32:0/2886060758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.205+0000 7f9239748640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d3c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f9239f49640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d3c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 -- 172.21.15.32:0/2886060758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 msgr2=0x565512d3c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d3c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 -- 172.21.15.32:0/2886060758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 msgr2=0x565512d3c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d3c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 -- 172.21.15.32:0/2886060758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565512dadae0 con 0x565512d389c0 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f9239f49640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d3c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923a74a640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f922c098950 tx=0x7f922c0cd9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c2f782db03a0eed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f92197fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c01d070 con 0x565512d389c0 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f922c09b3d0 con 0x565512d389c0 2024-04-23T18:19:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.206+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f922c0a0330 con 0x565512d389c0 2024-04-23T18:19:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f922c0b4040 con 0x565512d389c0 2024-04-23T18:19:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 10 client.25682 fetch_fsmap learned FSMap version 31 2024-04-23T18:19:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x565512daedf0 con 0x565512d389c0 2024-04-23T18:19:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f922c0a0330 con 0x565512d389c0 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f922c020060 con 0x565512d389c0 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 10 client.25682 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 20 client.25682 populate_metadata read hostname 'smithi032' 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512daf400 0x565512db1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 4 client.25682 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565512c9a200 con 0x565512daf400 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.207+0000 7f9239f49640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512daf400 0x565512db1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.208+0000 7f9239f49640 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512daf400 0x565512db1820 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec1e84e5df0230d server_cookie=f1d28fca2675b893 in_seq=0 out_seq=0 2024-04-23T18:19:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.208+0000 7f92197fa640 10 client.25682 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.230+0000 7f92197fa640 1 -- 172.21.15.32:0/2886060758 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565512c9a200 con 0x565512daf400 2024-04-23T18:19:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.230+0000 7f92197fa640 10 client.25682 handle_command_reply: tid=0 2024-04-23T18:19:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.230+0000 7f923ef9d740 1 client.25682 shutdown 2024-04-23T18:19:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.230+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 msgr2=0x565512d3ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.230+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f922c098950 tx=0x7f922c0cd9b0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 shutdown_connections 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x565512daf400 0x565512db1820 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565512d389c0 0x565512d3ce00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565512d2e100 0x565512d3c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 --2- 172.21.15.32:0/2886060758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565512d2d7e0 0x565512d3c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f92197fa640 0 client.25682 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 >> 172.21.15.32:0/2886060758 conn(0x565512d29cf0 msgr2=0x565512d28de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 shutdown_connections 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 1 -- 172.21.15.32:0/2886060758 wait complete. 2024-04-23T18:19:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.231+0000 7f923ef9d740 20 client.25682 trim_cache size 0 max 16384 2024-04-23T18:19:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:19:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:19:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:19:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:19:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 msgr2=0x7f921c0a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 0x7f921c0a6220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x565512d2b110 tx=0x7f9228004730 comp rx=0 tx=0).stop 2024-04-23T18:19:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 msgr2=0x7f923419e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f923419e890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f923406b340 tx=0x7f922c00cba0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 shutdown_connections 2024-04-23T18:19:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9234112c00 0x7f923419e890 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f921c0a3d60 0x7f921c0a6220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9234076da0 0x7f92341a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 --2- 172.21.15.32:0/2276662200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9234076480 0x7f92341a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 >> 172.21.15.32:0/2276662200 conn(0x7f92340ffa30 msgr2=0x7f9234101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 shutdown_connections 2024-04-23T18:19:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:19:48.233+0000 7f923af4b640 1 -- 172.21.15.32:0/2276662200 wait complete. 2024-04-23T18:19:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 260.0 s, 2662.8 tps, lat 37.797 ms stddev 60.510 2024-04-23T18:19:48.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:48 smithi064 ceph-mon[33062]: pgmap v144: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 689 op/s 2024-04-23T18:19:49.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:48 smithi032 ceph-mon[29221]: pgmap v144: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 689 op/s 2024-04-23T18:19:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:48 smithi040 ceph-mon[32626]: pgmap v144: 97 pgs: 97 active+clean; 2.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 689 op/s 2024-04-23T18:19:50.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:50 smithi064 ceph-mon[33062]: pgmap v145: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 MiB/s wr, 1.78k op/s 2024-04-23T18:19:50.922 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:50 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:51.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:50 smithi032 ceph-mon[29221]: pgmap v145: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 MiB/s wr, 1.78k op/s 2024-04-23T18:19:51.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:50 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:50 smithi040 ceph-mon[32626]: pgmap v145: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 MiB/s wr, 1.78k op/s 2024-04-23T18:19:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:50 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:52.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:52 smithi064 ceph-mon[33062]: pgmap v146: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 1.40k op/s 2024-04-23T18:19:52.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:52 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:53.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:52 smithi032 ceph-mon[29221]: pgmap v146: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 1.40k op/s 2024-04-23T18:19:53.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:52 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:52 smithi040 ceph-mon[32626]: pgmap v146: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 1.40k op/s 2024-04-23T18:19:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:52 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 265.0 s, 3325.6 tps, lat 30.145 ms stddev 34.085 2024-04-23T18:19:54.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:54 smithi064 ceph-mon[33062]: pgmap v147: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 MiB/s wr, 1.68k op/s 2024-04-23T18:19:55.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:54 smithi032 ceph-mon[29221]: pgmap v147: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 MiB/s wr, 1.68k op/s 2024-04-23T18:19:55.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:54 smithi040 ceph-mon[32626]: pgmap v147: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 MiB/s wr, 1.68k op/s 2024-04-23T18:19:56.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:56 smithi064 ceph-mon[33062]: pgmap v148: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 1.44k op/s 2024-04-23T18:19:57.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:56 smithi032 ceph-mon[29221]: pgmap v148: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 1.44k op/s 2024-04-23T18:19:57.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:56 smithi040 ceph-mon[32626]: pgmap v148: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 1.44k op/s 2024-04-23T18:19:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 270.0 s, 3099.6 tps, lat 32.190 ms stddev 33.919 2024-04-23T18:19:58.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:19:58 smithi064 ceph-mon[33062]: pgmap v149: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-23T18:19:59.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:19:58 smithi032 ceph-mon[29221]: pgmap v149: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-23T18:19:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:19:58 smithi040 ceph-mon[32626]: pgmap v149: 97 pgs: 97 active+clean; 2.9 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-23T18:20:00.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:00 smithi064 ceph-mon[33062]: pgmap v150: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 1.71k op/s 2024-04-23T18:20:00.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:00 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:20:01.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:00 smithi032 ceph-mon[29221]: pgmap v150: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 1.71k op/s 2024-04-23T18:20:01.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:00 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:20:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:00 smithi040 ceph-mon[32626]: pgmap v150: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 1.71k op/s 2024-04-23T18:20:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:00 smithi040 ceph-mon[32626]: overall HEALTH_OK 2024-04-23T18:20:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.373+0000 7f4bb88cf640 1 -- 172.21.15.32:0/4279251617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4101000 msgr2=0x7f4bb4104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.373+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/4279251617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4101000 0x7f4bb4104dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009900 tx=0x7f4ba802a410 comp rx=0 tx=0).stop 2024-04-23T18:20:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 -- 172.21.15.32:0/4279251617 shutdown_connections 2024-04-23T18:20:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/4279251617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb41054a0 0x7f4bb410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/4279251617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4101000 0x7f4bb4104dc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/4279251617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb4100650 0x7f4bb4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 -- 172.21.15.32:0/4279251617 >> 172.21.15.32:0/4279251617 conn(0x7f4bb4075d60 msgr2=0x7f4bb4076170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 -- 172.21.15.32:0/4279251617 shutdown_connections 2024-04-23T18:20:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 -- 172.21.15.32:0/4279251617 wait complete. 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 Processor -- start 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.374+0000 7f4bb88cf640 1 -- start start 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb41054a0 0x7f4bb4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47218/0 (socket says 172.21.15.32:47218) 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb259c640 1 -- 172.21.15.32:0/3280634906 learned_addr learned my addr 172.21.15.32:0/3280634906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 0x7f4bb419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410ffc0 con 0x7f4bb4100650 2024-04-23T18:20:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410fe40 con 0x7f4bb419ae00 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4110140 con 0x7f4bb41054a0 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb41054a0 0x7f4bb4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.375+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 0x7f4bb419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 -- 172.21.15.32:0/3280634906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb41054a0 msgr2=0x7f4bb4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb41054a0 0x7f4bb4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 -- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 msgr2=0x7f4bb419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 0x7f4bb419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb419b8f0 con 0x7f4bb4100650 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 0x7f4bb419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb259c640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400fad0 tx=0x7f4ba400ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4ba37fe640 1 -- 172.21.15.32:0/3280634906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400d760 con 0x7f4bb4100650 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4ba37fe640 1 -- 172.21.15.32:0/3280634906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba401a070 con 0x7f4bb4100650 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb419bb80 con 0x7f4bb4100650 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4ba37fe640 1 -- 172.21.15.32:0/3280634906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400a890 con 0x7f4bb4100650 2024-04-23T18:20:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.376+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb41a33a0 con 0x7f4bb4100650 2024-04-23T18:20:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 Processor -- start 2024-04-23T18:20:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 -- start start 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a8bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c5a39f0970 con 0x55c5a3a8b480 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c5a398d4a0 con 0x55c5a3a9f0a0 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.380+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c5a39ec930 con 0x55c5a3a8b860 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.381+0000 7f4bb1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.381+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.381+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47220/0 (socket says 172.21.15.32:47220) 2024-04-23T18:20:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.381+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/3594521901 learned_addr learned my addr 172.21.15.32:0/3594521901 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/3594521901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 msgr2=0x55c5a3a8bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:20:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a8bcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/3594521901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 msgr2=0x55c5a3aa1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/3594521901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5a3703210 con 0x55c5a3a8b480 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4ba37fe640 1 -- 172.21.15.32:0/3280634906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4ba4013050 con 0x7f4bb4100650 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011790 tx=0x7f4bac011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c9a207ce33aad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4ba0ff9640 1 -- 172.21.15.32:0/3594521901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac003c30 con 0x55c5a3a8b480 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4ba0ff9640 1 -- 172.21.15.32:0/3594521901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bac00fe50 con 0x55c5a3a8b480 2024-04-23T18:20:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.382+0000 7f4ba0ff9640 1 -- 172.21.15.32:0/3594521901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac004860 con 0x55c5a3a8b480 2024-04-23T18:20:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.383+0000 7f4ba37fe640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 0x7f4b9c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.383+0000 7f4bb259c640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 0x7f4b9c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.383+0000 7f4ba37fe640 1 -- 172.21.15.32:0/3280634906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4ba40d0090 con 0x7f4bb4100650 2024-04-23T18:20:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bb259c640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 0x7f4b9c0a6250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40cd980 tx=0x7f4ba4098440 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/3594521901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 msgr2=0x55c5a3a942b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011790 tx=0x7f4bac011c50 comp rx=0 tx=0).stop 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/3594521901 shutdown_connections 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a8bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/3594521901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a942b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/3594521901 >> 172.21.15.32:0/3594521901 conn(0x55c5a3a87b30 msgr2=0x55c5a3a87f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/3594521901 shutdown_connections 2024-04-23T18:20:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/3594521901 wait complete. 2024-04-23T18:20:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 -- start start 2024-04-23T18:20:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.384+0000 7f4bbb5e7740 1 Processor -- start 2024-04-23T18:20:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a91480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a919e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c5a3aa1c10 con 0x55c5a3a8b480 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c5a398b0b0 con 0x55c5a3a9f0a0 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c5a3aa1d90 con 0x55c5a3a8b860 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bbb5e7740 10 client.0 _ll_register_callbacks cb 0x55c5a3978d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47226/0 (socket says 172.21.15.32:47226) 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/1831741438 learned_addr learned my addr 172.21.15.32:0/1831741438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a91480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a919e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/1831741438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 msgr2=0x55c5a3a91480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a91480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 msgr2=0x55c5a3a919e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a919e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/1831741438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5a37cf620 con 0x55c5a3a8b480 2024-04-23T18:20:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.385+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4bac005b20 tx=0x7f4bac005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931896849a23b23a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.386+0000 7f4b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.386+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac02f060 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.386+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4bac003c30 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.386+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bac00fe50 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.387+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4bac01f040 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.387+0000 7f4bbb5e7740 10 client.16065 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.387+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c5a3aadb80 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bac01a8f0 con 0x55c5a3a8b480 2024-04-23T18:20:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f4bac018070 con 0x55c5a3a8b480 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 10 client.16065 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 20 client.16065 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3aaef30 0x55c5a3ab1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 4 client.16065 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4bac00fe50 con 0x55c5a3aaef30 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.388+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3aaef30 0x55c5a3ab1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.389+0000 7f4b99ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.389+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3aaef30 0x55c5a3ab1320 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31849e002d306217 server_cookie=c828be50b52d9292 in_seq=0 out_seq=0 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.389+0000 7f4b99ffb640 1 -- 172.21.15.32:0/1831741438 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4bac00fe50 con 0x55c5a3aaef30 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.389+0000 7f4b99ffb640 10 client.16065 handle_command_reply: tid=0 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.389+0000 7f4bbb5e7740 1 client.16065 shutdown 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 msgr2=0x55c5a3a94140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4bac005b20 tx=0x7f4bac005a50 comp rx=0 tx=0).stop 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 shutdown_connections 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3aaef30 0x55c5a3ab1320 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a919e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3a91480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/1831741438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3a94140 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 >> 172.21.15.32:0/1831741438 conn(0x55c5a3a87b30 msgr2=0x55c5a3a8d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4b99ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 shutdown_connections 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/1831741438 wait complete. 2024-04-23T18:20:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.390+0000 7f4bbb5e7740 20 client.16065 trim_cache size 0 max 16384 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 Processor -- start 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 -- start start 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 0x55c5a3a90a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a93c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41ae3c0 con 0x55c5a3a8b480 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41abc60 con 0x55c5a3a8b860 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41a9ad0 con 0x55c5a3a9f0a0 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bb1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 0x55c5a3a90a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47230/0 (socket says 172.21.15.32:47230) 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bb2d9d640 1 -- 172.21.15.32:0/2466913739 learned_addr learned my addr 172.21.15.32:0/2466913739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.415+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a93c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2466913739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a9f0a0 msgr2=0x55c5a3a93c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a93c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2466913739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 msgr2=0x55c5a3aaaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2466913739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5a3a8fa30 con 0x55c5a3a8b860 2024-04-23T18:20:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.416+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.418+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 0x55c5a3a90a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ba802a930 tx=0x7f4ba8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7905f42f87aea89d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.418+0000 7f4b9bfff640 1 -- 172.21.15.32:0/2466913739 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8035040 con 0x55c5a3a8b860 2024-04-23T18:20:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.418+0000 7f4b9bfff640 1 -- 172.21.15.32:0/2466913739 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba8046830 con 0x55c5a3a8b860 2024-04-23T18:20:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.418+0000 7f4b9bfff640 1 -- 172.21.15.32:0/2466913739 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80053c0 con 0x55c5a3a8b860 2024-04-23T18:20:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2466913739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 msgr2=0x55c5a3a90a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 0x55c5a3a90a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ba802a930 tx=0x7f4ba8004610 comp rx=0 tx=0).stop 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2466913739 shutdown_connections 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3a93c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b860 0x55c5a3a90a20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2466913739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a8b480 0x55c5a3aaaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2466913739 >> 172.21.15.32:0/2466913739 conn(0x55c5a3a87b30 msgr2=0x55c5a3a8fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2466913739 shutdown_connections 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2466913739 wait complete. 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 -- start start 2024-04-23T18:20:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.419+0000 7f4bbb5e7740 1 Processor -- start 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b480 0x55c5a3b0d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41aa120 con 0x55c5a3a9f0a0 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41ab590 con 0x55c5a3a8b480 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb41aa7f0 con 0x55c5a3a8b860 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bbb5e7740 10 client.0 _ll_register_callbacks cb 0x7f4bac0041e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bb1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b480 0x55c5a3b0d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bb1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:37540/0 (socket says 172.21.15.32:37540) 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2781446280 learned_addr learned my addr 172.21.15.32:0/2781446280 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.420+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b480 msgr2=0x55c5a3b0d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b480 0x55c5a3b0d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2781446280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a9f0a0 msgr2=0x55c5a3aa4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 -- 172.21.15.32:0/2781446280 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5a3aa49e0 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bb1d9b640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80041f0 tx=0x7f4ba80465b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=986099c2fa9da336 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8035070 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4ba8046bf0 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.421+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ba8046830 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ba8046e30 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 10 client.25681 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c5a3b14c10 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba80054f0 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f4ba80035f0 con 0x55c5a3a8b860 2024-04-23T18:20:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 10 client.25681 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 20 client.25681 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3b15360 0x55c5a3b17780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 4 client.25681 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.422+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4ba8046830 con 0x55c5a3b15360 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.424+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3b15360 0x55c5a3b17780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.425+0000 7f4b99ffb640 10 client.25681 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.425+0000 7f4bb2d9d640 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3b15360 0x55c5a3b17780 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af11a2e8a19b701 server_cookie=53c0a994cc79a094 in_seq=0 out_seq=0 2024-04-23T18:20:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.437+0000 7f4b99ffb640 1 -- 172.21.15.32:0/2781446280 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4ba8046830 con 0x55c5a3b15360 2024-04-23T18:20:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.437+0000 7f4b99ffb640 10 client.25681 handle_command_reply: tid=0 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.437+0000 7f4bbb5e7740 1 client.25681 shutdown 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.437+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 msgr2=0x55c5a3b0da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80041f0 tx=0x7f4ba80465b0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 shutdown_connections 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c5a3b15360 0x55c5a3b17780 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c5a3a9f0a0 0x55c5a3aa4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5a3a8b860 0x55c5a3b0da60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 --2- 172.21.15.32:0/2781446280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5a3a8b480 0x55c5a3b0d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4b99ffb640 0 client.25681 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 >> 172.21.15.32:0/2781446280 conn(0x55c5a3a87b30 msgr2=0x55c5a3a86420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 shutdown_connections 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 1 -- 172.21.15.32:0/2781446280 wait complete. 2024-04-23T18:20:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.438+0000 7f4bbb5e7740 20 client.25681 trim_cache size 0 max 16384 2024-04-23T18:20:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:20:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:20:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:20:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 msgr2=0x7f4b9c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 0x7f4b9c0a6250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40cd980 tx=0x7f4ba4098440 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 msgr2=0x7f4bb4196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400fad0 tx=0x7f4ba400ff90 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 shutdown_connections 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bb419ae00 0x7f4bb419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bb41054a0 0x7f4bb4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4b9c0a3d90 0x7f4b9c0a6250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 --2- 172.21.15.32:0/3280634906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bb4100650 0x7f4bb4196540 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.439+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 >> 172.21.15.32:0/3280634906 conn(0x7f4bb4075d60 msgr2=0x7f4bb40fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.440+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 shutdown_connections 2024-04-23T18:20:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:01.440+0000 7f4bb88cf640 1 -- 172.21.15.32:0/3280634906 wait complete. 2024-04-23T18:20:02.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:02 smithi064 ceph-mon[33062]: pgmap v151: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.3 MiB/s wr, 613 op/s 2024-04-23T18:20:03.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:02 smithi032 ceph-mon[29221]: pgmap v151: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.3 MiB/s wr, 613 op/s 2024-04-23T18:20:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:02 smithi040 ceph-mon[32626]: pgmap v151: 97 pgs: 97 active+clean; 2.9 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.3 MiB/s wr, 613 op/s 2024-04-23T18:20:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 275.0 s, 2285.6 tps, lat 43.781 ms stddev 90.679 2024-04-23T18:20:04.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:04 smithi064 ceph-mon[33062]: pgmap v152: 97 pgs: 97 active+clean; 2.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 26 MiB/s wr, 1.86k op/s 2024-04-23T18:20:05.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:04 smithi032 ceph-mon[29221]: pgmap v152: 97 pgs: 97 active+clean; 2.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 26 MiB/s wr, 1.86k op/s 2024-04-23T18:20:05.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:04 smithi040 ceph-mon[32626]: pgmap v152: 97 pgs: 97 active+clean; 2.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 26 MiB/s wr, 1.86k op/s 2024-04-23T18:20:05.905 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-23T18:20:05.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.073+0000 7f6342322640 1 -- 172.21.15.32:0/898865045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c1021a0 msgr2=0x7f633c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.073+0000 7f6342322640 1 --2- 172.21.15.32:0/898865045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c1021a0 0x7f633c102580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f632c009950 tx=0x7f632c02a440 comp rx=0 tx=0).stop 2024-04-23T18:20:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.074+0000 7f6342322640 1 -- 172.21.15.32:0/898865045 shutdown_connections 2024-04-23T18:20:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.074+0000 7f6342322640 1 --2- 172.21.15.32:0/898865045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c107270 0x7f633c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.074+0000 7f6342322640 1 --2- 172.21.15.32:0/898865045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c102b50 0x7f633c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.074+0000 7f6342322640 1 --2- 172.21.15.32:0/898865045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c1021a0 0x7f633c102580 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.074+0000 7f6342322640 1 -- 172.21.15.32:0/898865045 >> 172.21.15.32:0/898865045 conn(0x7f633c0fda40 msgr2=0x7f633c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.075+0000 7f6342322640 1 -- 172.21.15.32:0/898865045 shutdown_connections 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.075+0000 7f6342322640 1 -- 172.21.15.32:0/898865045 wait complete. 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 Processor -- start 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 -- start start 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 0x7f633c1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 0x7f633c10c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f633c112220 con 0x7f633c107270 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f633c1120a0 con 0x7f633c10bee0 2024-04-23T18:20:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.076+0000 7f6342322640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f633c1123a0 con 0x7f633c1021a0 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 0x7f633c1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47256/0 (socket says 172.21.15.32:47256) 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 -- 172.21.15.32:0/866393924 learned_addr learned my addr 172.21.15.32:0/866393924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f63408bf640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 0x7f633c10c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 -- 172.21.15.32:0/866393924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 msgr2=0x7f633c1a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 0x7f633c1a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 -- 172.21.15.32:0/866393924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 msgr2=0x7f633c10c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 0x7f633c10c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633b7fe640 1 -- 172.21.15.32:0/866393924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633c10cac0 con 0x7f633c107270 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f633bfff640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 0x7f633c1a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.077+0000 7f63408bf640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 0x7f633c10c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.078+0000 7f633b7fe640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f633000d740 tx=0x7f633000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.078+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330017070 con 0x7f633c107270 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.078+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f633000de50 con 0x7f633c107270 2024-04-23T18:20:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.078+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330004ce0 con 0x7f633c107270 2024-04-23T18:20:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.078+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633c1ab890 con 0x7f633c107270 2024-04-23T18:20:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.079+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633c1abc40 con 0x7f633c107270 2024-04-23T18:20:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.082+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6330004e80 con 0x7f633c107270 2024-04-23T18:20:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.082+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f633c103670 con 0x7f633c107270 2024-04-23T18:20:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.083+0000 7f63397fa640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 0x7f63240a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.083+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f63300d0110 con 0x7f633c107270 2024-04-23T18:20:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.083+0000 7f633bfff640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 0x7f63240a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.085+0000 7f633bfff640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 0x7f63240a6330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f632c006010 tx=0x7f632c000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.097+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f633009c7a0 con 0x7f633c107270 2024-04-23T18:20:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.510+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f633c102b50 con 0x7f633c107270 2024-04-23T18:20:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.511+0000 7f63397fa640 1 -- 172.21.15.32:0/866393924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11852 (secure 0 0 0) 0x7f633009c0a0 con 0x7f633c107270 2024-04-23T18:20:06.505 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:20:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:19:50.863737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:20:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-23T18:20:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.518+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 msgr2=0x7f63240a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.518+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 0x7f63240a6330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f632c006010 tx=0x7f632c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.518+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 msgr2=0x7f633c1a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.518+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f633000d740 tx=0x7f633000dc00 comp rx=0 tx=0).stop 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 shutdown_connections 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c10bee0 0x7f633c10c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f63240a3e70 0x7f63240a6330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f633c107270 0x7f633c1a3eb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 --2- 172.21.15.32:0/866393924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f633c1021a0 0x7f633c1a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.519+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 >> 172.21.15.32:0/866393924 conn(0x7f633c0fda40 msgr2=0x7f633c0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.520+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 shutdown_connections 2024-04-23T18:20:06.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.522+0000 7f6342322640 1 -- 172.21.15.32:0/866393924 wait complete. 2024-04-23T18:20:06.538 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-23T18:20:06.538 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-23T18:20:06.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.710+0000 7f698a81e640 1 -- 172.21.15.32:0/2919177450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6984107290 msgr2=0x7f698410ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.710+0000 7f698a81e640 1 --2- 172.21.15.32:0/2919177450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6984107290 0x7f698410ee60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f697c009ca0 tx=0x7f697c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 -- 172.21.15.32:0/2919177450 shutdown_connections 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 --2- 172.21.15.32:0/2919177450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6984107290 0x7f698410ee60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 --2- 172.21.15.32:0/2919177450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984102b70 0x7f6984106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 --2- 172.21.15.32:0/2919177450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 -- 172.21.15.32:0/2919177450 >> 172.21.15.32:0/2919177450 conn(0x7f69840fda00 msgr2=0x7f69840ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 -- 172.21.15.32:0/2919177450 shutdown_connections 2024-04-23T18:20:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 -- 172.21.15.32:0/2919177450 wait complete. 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.712+0000 7f698a81e640 1 Processor -- start 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 -- start start 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 0x7f6984196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f698419ae60 0x7f698419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6984110030 con 0x7f69841021c0 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f698410feb0 con 0x7f698419ae60 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f698a81e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69841101b0 con 0x7f6984107290 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f6983fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f6983fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50896/0 (socket says 172.21.15.32:50896) 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f6983fff640 1 -- 172.21.15.32:0/3402605461 learned_addr learned my addr 172.21.15.32:0/3402605461 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f6988dbb640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f698419ae60 0x7f698419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.713+0000 7f69837fe640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 0x7f6984196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 -- 172.21.15.32:0/3402605461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 msgr2=0x7f6984196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 0x7f6984196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 -- 172.21.15.32:0/3402605461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f698419ae60 msgr2=0x7f698419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f698419ae60 0x7f698419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 -- 172.21.15.32:0/3402605461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698419b950 con 0x7f69841021c0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f69837fe640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 0x7f6984196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.714+0000 7f6983fff640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f697400e9f0 tx=0x7f697400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.715+0000 7f69817fa640 1 -- 172.21.15.32:0/3402605461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697400cd80 con 0x7f69841021c0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.715+0000 7f69817fa640 1 -- 172.21.15.32:0/3402605461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69740108e0 con 0x7f69841021c0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.715+0000 7f69817fa640 1 -- 172.21.15.32:0/3402605461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974018980 con 0x7f69841021c0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.715+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698419bc40 con 0x7f69841021c0 2024-04-23T18:20:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.715+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69841a3320 con 0x7f69841021c0 2024-04-23T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.719+0000 7f69817fa640 1 -- 172.21.15.32:0/3402605461 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6974010430 con 0x7f69841021c0 2024-04-23T18:20:06.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.719+0000 7f698d60d740 1 Processor -- start 2024-04-23T18:20:06.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.719+0000 7f698d60d740 1 -- start start 2024-04-23T18:20:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df136360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df1368a0 0x55b9df138c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b9df088370 con 0x55b9df12c1d0 2024-04-23T18:20:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b9df0252e0 con 0x55b9df1368a0 2024-04-23T18:20:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f698d60d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b9df084330 con 0x55b9df12b8b0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.720+0000 7f69817fa640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 0x7f696c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f69817fa640 1 -- 172.21.15.32:0/3402605461 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f69740d0a30 con 0x7f69841021c0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f6988dbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f6988dbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53012/0 (socket says 172.21.15.32:53012) 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f6988dbb640 1 -- 172.21.15.32:0/686396751 learned_addr learned my addr 172.21.15.32:0/686396751 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f6988dbb640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 0x7f696c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f6983fff640 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.721+0000 7f69837fe640 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df1368a0 0x55b9df138c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 -- 172.21.15.32:0/686396751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df1368a0 msgr2=0x55b9df138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df1368a0 0x55b9df138c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 -- 172.21.15.32:0/686396751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 msgr2=0x55b9df136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 -- 172.21.15.32:0/686396751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9dee66d80 con 0x55b9df12b8b0 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f697c02aeb0 tx=0x7f697c031e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9997ee8b24febfeb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f696a7fc640 1 -- 172.21.15.32:0/686396751 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c004d40 con 0x55b9df12b8b0 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f696a7fc640 1 -- 172.21.15.32:0/686396751 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c04f070 con 0x55b9df12b8b0 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f696a7fc640 1 -- 172.21.15.32:0/686396751 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c0046f0 con 0x55b9df12b8b0 2024-04-23T18:20:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.722+0000 7f6988dbb640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 0x7f696c0a6320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f697c04e000 tx=0x7f697c04c000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 -- 172.21.15.32:0/686396751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 msgr2=0x55b9df12bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f697c02aeb0 tx=0x7f697c031e10 comp rx=0 tx=0).stop 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 -- 172.21.15.32:0/686396751 shutdown_connections 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df1368a0 0x55b9df138c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 --2- 172.21.15.32:0/686396751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df12b8b0 0x55b9df12bc90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.723+0000 7f698d60d740 1 -- 172.21.15.32:0/686396751 >> 172.21.15.32:0/686396751 conn(0x55b9df127bd0 msgr2=0x55b9df127fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.730+0000 7f698d60d740 1 -- 172.21.15.32:0/686396751 shutdown_connections 2024-04-23T18:20:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.731+0000 7f698d60d740 1 -- 172.21.15.32:0/686396751 wait complete. 2024-04-23T18:20:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.731+0000 7f698d60d740 1 -- start start 2024-04-23T18:20:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.731+0000 7f698d60d740 1 Processor -- start 2024-04-23T18:20:06.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df133820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df1342a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b9df139510 con 0x55b9df12c1d0 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b9df022fb0 con 0x55b9df12b8b0 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b9df139690 con 0x55b9df1368a0 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f698d60d740 10 client.0 _ll_register_callbacks cb 0x55b9df0229a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f69837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f69837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50922/0 (socket says 172.21.15.32:50922) 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.735+0000 7f69837fe640 1 -- 172.21.15.32:0/1319638335 learned_addr learned my addr 172.21.15.32:0/1319638335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.736+0000 7f6988dbb640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df1342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 -- 172.21.15.32:0/1319638335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 msgr2=0x55b9df1342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df1342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 -- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 msgr2=0x55b9df133820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:20:06.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df133820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 -- 172.21.15.32:0/1319638335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9deec7510 con 0x55b9df12c1d0 2024-04-23T18:20:06.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69837fe640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f69780129e0 tx=0x7f6978012ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b136680a144c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.739+0000 7f69637fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.740+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978004030 con 0x55b9df12c1d0 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.740+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697c04f070 con 0x55b9df12c1d0 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.740+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f69780041d0 con 0x55b9df12c1d0 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.740+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f69780043d0 con 0x55b9df12c1d0 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.741+0000 7f698d60d740 10 client.16089 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.741+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b9ded5a670 con 0x55b9df12c1d0 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c04f070 con 0x55b9df12c1d0 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f6978003cc0 con 0x55b9df12c1d0 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 10 client.16089 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 10 client.16089 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 20 client.16089 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df123350 0x55b9df1257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 4 client.16089 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.744+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9df095d90 con 0x55b9df123350 2024-04-23T18:20:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.746+0000 7f6983fff640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df123350 0x55b9df1257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.748+0000 7f6983fff640 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df123350 0x55b9df1257f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde281eccbf8a579 server_cookie=3f96f1981f43a41c in_seq=0 out_seq=0 2024-04-23T18:20:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.749+0000 7f69637fe640 10 client.16089 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.749+0000 7f69637fe640 1 -- 172.21.15.32:0/1319638335 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9df095d90 con 0x55b9df123350 2024-04-23T18:20:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.749+0000 7f69637fe640 10 client.16089 handle_command_reply: tid=0 2024-04-23T18:20:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.749+0000 7f698d60d740 1 client.16089 shutdown 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.752+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 msgr2=0x55b9df133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.752+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f69780129e0 tx=0x7f6978012ea0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 shutdown_connections 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df123350 0x55b9df1257f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df1342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df133d60 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 --2- 172.21.15.32:0/1319638335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df133820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 >> 172.21.15.32:0/1319638335 conn(0x55b9df127bd0 msgr2=0x55b9df11e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.753+0000 7f69637fe640 0 client.16089 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:06.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.754+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 shutdown_connections 2024-04-23T18:20:06.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.755+0000 7f698d60d740 1 -- 172.21.15.32:0/1319638335 wait complete. 2024-04-23T18:20:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.755+0000 7f698d60d740 20 client.16089 trim_cache size 0 max 16384 2024-04-23T18:20:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.790+0000 7f698d60d740 1 Processor -- start 2024-04-23T18:20:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.790+0000 7f698d60d740 1 -- start start 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df1423d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69841ae250 con 0x55b9df12c1d0 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69841abaf0 con 0x55b9df12b8b0 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f698d60d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69841a9940 con 0x55b9df1368a0 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f69837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f6983fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df1423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f69837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50926/0 (socket says 172.21.15.32:50926) 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.791+0000 7f69837fe640 1 -- 172.21.15.32:0/68238928 learned_addr learned my addr 172.21.15.32:0/68238928 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 -- 172.21.15.32:0/68238928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 msgr2=0x55b9df158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 -- 172.21.15.32:0/68238928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 msgr2=0x55b9df1423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df1423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 -- 172.21.15.32:0/68238928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9df128550 con 0x55b9df12c1d0 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f6983fff640 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df1423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f69837fe640 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6978012fd0 tx=0x7f69780126d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c5da16c9ce9de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f6969ffb640 1 -- 172.21.15.32:0/68238928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978026750 con 0x55b9df12c1d0 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f6969ffb640 1 -- 172.21.15.32:0/68238928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697800fe60 con 0x55b9df12c1d0 2024-04-23T18:20:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.792+0000 7f6969ffb640 1 -- 172.21.15.32:0/68238928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978010430 con 0x55b9df12c1d0 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 -- 172.21.15.32:0/68238928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 msgr2=0x55b9df1589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6978012fd0 tx=0x7f69780126d0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 -- 172.21.15.32:0/68238928 shutdown_connections 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12c1d0 0x55b9df1589c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 --2- 172.21.15.32:0/68238928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12b8b0 0x55b9df1423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.793+0000 7f698d60d740 1 -- 172.21.15.32:0/68238928 >> 172.21.15.32:0/68238928 conn(0x55b9df127bd0 msgr2=0x55b9df11e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.794+0000 7f698d60d740 1 -- 172.21.15.32:0/68238928 shutdown_connections 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.794+0000 7f698d60d740 1 -- 172.21.15.32:0/68238928 wait complete. 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.794+0000 7f698d60d740 1 -- start start 2024-04-23T18:20:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 Processor -- start 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 0x55b9df1398a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 0x55b9df139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df13a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69841a9fb0 con 0x55b9df12b8b0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69841ab420 con 0x55b9df12c1d0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69841aa680 con 0x55b9df1368a0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f698d60d740 10 client.0 _ll_register_callbacks cb 0x7f6978012620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f6983fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 0x55b9df1398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f6983fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 0x55b9df1398a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50936/0 (socket says 172.21.15.32:50936) 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f6983fff640 1 -- 172.21.15.32:0/3053890017 learned_addr learned my addr 172.21.15.32:0/3053890017 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f69837fe640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 0x55b9df139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.795+0000 7f6988dbb640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df13a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 -- 172.21.15.32:0/3053890017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 msgr2=0x55b9df13a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df13a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 -- 172.21.15.32:0/3053890017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 msgr2=0x55b9df1398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 0x55b9df1398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 -- 172.21.15.32:0/3053890017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9df1b5220 con 0x55b9df12c1d0 2024-04-23T18:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69837fe640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 0x55b9df139de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55b9df11ae50 tx=0x7f6978026d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eca1c5272ec213c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.796+0000 7f69637fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697800fbc0 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6978018070 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697800fe60 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6978005040 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f698d60d740 10 client.25718 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.797+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b9df1b6440 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697800fe60 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f69780049f0 con 0x55b9df12c1d0 2024-04-23T18:20:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 10 client.25718 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 10 client.25718 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 20 client.25718 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df1b6a60 0x55b9df1b8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 4 client.25718 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x55b9df08fc00 con 0x55b9df1b6a60 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.798+0000 7f6983fff640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df1b6a60 0x55b9df1b8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.799+0000 7f6983fff640 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df1b6a60 0x55b9df1b8e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56e3327e6084b3ca server_cookie=c60ce1f6d75dd064 in_seq=0 out_seq=0 2024-04-23T18:20:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:06.799+0000 7f69637fe640 10 client.25718 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:06.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.003+0000 7f69637fe640 1 -- 172.21.15.32:0/3053890017 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b9df08fc00 con 0x55b9df1b6a60 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.003+0000 7f69637fe640 10 client.25718 handle_command_reply: tid=0 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.003+0000 7f698d60d740 1 client.25718 shutdown 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 msgr2=0x55b9df139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 0x55b9df139de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55b9df11ae50 tx=0x7f6978026d90 comp rx=0 tx=0).stop 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 shutdown_connections 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b9df1b6a60 0x55b9df1b8e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b9df1368a0 0x55b9df13a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b9df12c1d0 0x55b9df139de0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 --2- 172.21.15.32:0/3053890017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9df12b8b0 0x55b9df1398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f69637fe640 0 client.25718 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 >> 172.21.15.32:0/3053890017 conn(0x55b9df127bd0 msgr2=0x55b9df11dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 shutdown_connections 2024-04-23T18:20:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 1 -- 172.21.15.32:0/3053890017 wait complete. 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698d60d740 20 client.25718 trim_cache size 0 max 16384 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 33, 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 14, 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "8224a456": { 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 14, 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:20:06.999 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 70.0, 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:07.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:07.001 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:20:07.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 msgr2=0x7f696c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:07.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 0x7f696c0a6320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f697c04e000 tx=0x7f697c04c000 comp rx=0 tx=0).stop 2024-04-23T18:20:07.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 msgr2=0x7f69841965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f697400e9f0 tx=0x7f697400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 shutdown_connections 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f698419ae60 0x7f698419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6984107290 0x7f6984196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f696c0a3e60 0x7f696c0a6320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 --2- 172.21.15.32:0/3402605461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69841021c0 0x7f69841965f0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.004+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 >> 172.21.15.32:0/3402605461 conn(0x7f69840fda00 msgr2=0x7f69840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.005+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 shutdown_connections 2024-04-23T18:20:07.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:07.005+0000 7f698a81e640 1 -- 172.21.15.32:0/3402605461 wait complete. 2024-04-23T18:20:07.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:06 smithi032 ceph-mon[29221]: pgmap v153: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.83k op/s 2024-04-23T18:20:07.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:06 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/866393924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:07.009 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8224a456', quiesce took 1.1 seconds. Will release after: 10 2024-04-23T18:20:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:06 smithi040 ceph-mon[32626]: pgmap v153: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.83k op/s 2024-04-23T18:20:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:06 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/866393924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:06 smithi064 ceph-mon[33062]: pgmap v153: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.83k op/s 2024-04-23T18:20:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:06 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/866393924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:08.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 280.0 s, 1829.8 tps, lat 32.347 ms stddev 40.319 2024-04-23T18:20:08.974 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:08 smithi064 ceph-mon[33062]: pgmap v154: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.80k op/s 2024-04-23T18:20:09.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:08 smithi032 ceph-mon[29221]: pgmap v154: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.80k op/s 2024-04-23T18:20:09.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:08 smithi040 ceph-mon[32626]: pgmap v154: 97 pgs: 97 active+clean; 3.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 27 MiB/s wr, 1.80k op/s 2024-04-23T18:20:10.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:10 smithi040 ceph-mon[32626]: pgmap v155: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 43 MiB/s wr, 2.24k op/s 2024-04-23T18:20:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:10 smithi032 ceph-mon[29221]: pgmap v155: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 43 MiB/s wr, 2.24k op/s 2024-04-23T18:20:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:10 smithi064 ceph-mon[33062]: pgmap v155: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 43 MiB/s wr, 2.24k op/s 2024-04-23T18:20:13.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:12 smithi032 ceph-mon[29221]: pgmap v156: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:13.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:12 smithi040 ceph-mon[32626]: pgmap v156: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:13.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:12 smithi064 ceph-mon[33062]: pgmap v156: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:13.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 285.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:20:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 -- 172.21.15.32:0/1358314704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0101350 msgr2=0x7fd9e0101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/1358314704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0101350 0x7fd9e0101730 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009930 tx=0x7fd9d002a420 comp rx=0 tx=0).stop 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 -- 172.21.15.32:0/1358314704 shutdown_connections 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/1358314704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 0x7fd9e010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/1358314704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101d00 0x7fd9e0105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/1358314704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0101350 0x7fd9e0101730 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 -- 172.21.15.32:0/1358314704 >> 172.21.15.32:0/1358314704 conn(0x7fd9e0077d10 msgr2=0x7fd9e0078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 -- 172.21.15.32:0/1358314704 shutdown_connections 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.543+0000 7fd9e63bf640 1 -- 172.21.15.32:0/1358314704 wait complete. 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 Processor -- start 2024-04-23T18:20:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 -- start start 2024-04-23T18:20:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101350 0x7fd9e0199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 0x7fd9e0199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01110e0 con 0x7fd9e0199ff0 2024-04-23T18:20:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e0110f60 con 0x7fd9e0101350 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.544+0000 7fd9e63bf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e0111260 con 0x7fd9e0106220 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50958/0 (socket says 172.21.15.32:50958) 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 -- 172.21.15.32:0/2200181307 learned_addr learned my addr 172.21.15.32:0/2200181307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 0x7fd9e0199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9dffff640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101350 0x7fd9e0199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 -- 172.21.15.32:0/2200181307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 msgr2=0x7fd9e0199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 0x7fd9e0199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 -- 172.21.15.32:0/2200181307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101350 msgr2=0x7fd9e0199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101350 0x7fd9e0199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 -- 172.21.15.32:0/2200181307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e01a4d70 con 0x7fd9e0199ff0 2024-04-23T18:20:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e495c640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800cce0 tx=0x7fd9d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9dd7fa640 1 -- 172.21.15.32:0/2200181307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800f450 con 0x7fd9e0199ff0 2024-04-23T18:20:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9dd7fa640 1 -- 172.21.15.32:0/2200181307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d800faa0 con 0x7fd9e0199ff0 2024-04-23T18:20:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9dd7fa640 1 -- 172.21.15.32:0/2200181307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8018b70 con 0x7fd9e0199ff0 2024-04-23T18:20:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e01a5060 con 0x7fd9e0199ff0 2024-04-23T18:20:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.545+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e01a5560 con 0x7fd9e0199ff0 2024-04-23T18:20:14.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9dd7fa640 1 -- 172.21.15.32:0/2200181307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd9d8018d10 con 0x7fd9e0199ff0 2024-04-23T18:20:14.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 Processor -- start 2024-04-23T18:20:14.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 -- start start 2024-04-23T18:20:14.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb6fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbabad4970 con 0x55dbabb6ff40 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbaba714a0 con 0x55dbabb6f620 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.548+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbabad0930 con 0x55dbabb7a800 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dd7fa640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 0x7fd9c80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50972/0 (socket says 172.21.15.32:50972) 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 -- 172.21.15.32:0/3488684257 learned_addr learned my addr 172.21.15.32:0/3488684257 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9e495c640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 0x7fd9c80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb6fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 -- 172.21.15.32:0/3488684257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 msgr2=0x55dbabb7cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dd7fa640 1 -- 172.21.15.32:0/2200181307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd9d80d0b10 con 0x7fd9e0199ff0 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 -- 172.21.15.32:0/3488684257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 msgr2=0x55dbabb6fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb6fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 -- 172.21.15.32:0/3488684257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbab7e7210 con 0x55dbabb6ff40 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb6fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9dffff640 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009930 tx=0x7fd9d0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6af700bf50f3fb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9e495c640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 0x7fd9c80a61d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80ce360 tx=0x7fd9d8087630 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.549+0000 7fd9c67fc640 1 -- 172.21.15.32:0/3488684257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0035040 con 0x55dbabb6ff40 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9c67fc640 1 -- 172.21.15.32:0/3488684257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d00316a0 con 0x55dbabb6ff40 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9c67fc640 1 -- 172.21.15.32:0/3488684257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d003f7e0 con 0x55dbabb6ff40 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 -- 172.21.15.32:0/3488684257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 msgr2=0x55dbabb7a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009930 tx=0x7fd9d0005fd0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 -- 172.21.15.32:0/3488684257 shutdown_connections 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7a2c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/3488684257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb6fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 -- 172.21.15.32:0/3488684257 >> 172.21.15.32:0/3488684257 conn(0x55dbabb6bb30 msgr2=0x55dbabb6bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 -- 172.21.15.32:0/3488684257 shutdown_connections 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.550+0000 7fd9e91b3740 1 -- 172.21.15.32:0/3488684257 wait complete. 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 -- start start 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 Processor -- start 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6ff40 0x55dbabb77410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbabb7d3d0 con 0x55dbabb6f620 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbaba6f0b0 con 0x55dbabb6ff40 2024-04-23T18:20:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbabb7d550 con 0x55dbabb7a800 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9e91b3740 10 client.0 _ll_register_callbacks cb 0x55dbaba5cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50984/0 (socket says 172.21.15.32:50984) 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.551+0000 7fd9df7fe640 1 -- 172.21.15.32:0/1102932935 learned_addr learned my addr 172.21.15.32:0/1102932935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6ff40 0x55dbabb77410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb77950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 -- 172.21.15.32:0/1102932935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 msgr2=0x55dbabb77950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb77950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 -- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6ff40 msgr2=0x55dbabb77410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6ff40 0x55dbabb77410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 -- 172.21.15.32:0/1102932935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbab8b3620 con 0x55dbabb6f620 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4004400 tx=0x7fd9d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03ff8a1cebbd8df server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:14.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4005220 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd9d40053c0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9d00316a0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd9d40055c0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9e91b3740 10 client.16107 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dbabb892b0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.552+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d00316a0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fd9d4010ac0 con 0x55dbabb6f620 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 10 client.16107 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 20 client.16107 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabb8a770 0x55dbabb8cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 4 client.16107 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbabae2390 con 0x55dbabb8a770 2024-04-23T18:20:14.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabb8a770 0x55dbabb8cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabb8a770 0x55dbabb8cb80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c3638150660ac50 server_cookie=792ee38605fd1b48 in_seq=0 out_seq=0 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.553+0000 7fd9bf7fe640 10 client.16107 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.554+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1102932935 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbabae2390 con 0x55dbabb8a770 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.554+0000 7fd9bf7fe640 10 client.16107 handle_command_reply: tid=0 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.554+0000 7fd9e91b3740 1 client.16107 shutdown 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 msgr2=0x55dbabb7a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4004400 tx=0x7fd9d4004730 comp rx=0 tx=0).stop 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 shutdown_connections 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabb8a770 0x55dbabb8cb80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb77950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6ff40 0x55dbabb77410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1102932935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6f620 0x55dbabb7a120 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9bf7fe640 0 client.16107 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 >> 172.21.15.32:0/1102932935 conn(0x55dbabb6bb30 msgr2=0x55dbabb718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 shutdown_connections 2024-04-23T18:20:14.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1102932935 wait complete. 2024-04-23T18:20:14.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.555+0000 7fd9e91b3740 20 client.16107 trim_cache size 0 max 16384 2024-04-23T18:20:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 Processor -- start 2024-04-23T18:20:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 -- start start 2024-04-23T18:20:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb6f620 0x55dbabb86290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 0x55dbabb92490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01ae180 con 0x55dbabb6ff40 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01aba20 con 0x55dbabb7a800 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01a9890 con 0x55dbabb6f620 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50996/0 (socket says 172.21.15.32:50996) 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 -- 172.21.15.32:0/4197435718 learned_addr learned my addr 172.21.15.32:0/4197435718 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 0x55dbabb92490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb6f620 0x55dbabb86290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 -- 172.21.15.32:0/4197435718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb6f620 msgr2=0x55dbabb86290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb6f620 0x55dbabb86290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 -- 172.21.15.32:0/4197435718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 msgr2=0x55dbabb92490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 0x55dbabb92490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9dffff640 1 -- 172.21.15.32:0/4197435718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbabb6c700 con 0x55dbabb6ff40 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.570+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 0x55dbabb92490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9dffff640 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009930 tx=0x7fd9d0032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05e7d19d5584036 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9c5ffb640 1 -- 172.21.15.32:0/4197435718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0059070 con 0x55dbabb6ff40 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9c5ffb640 1 -- 172.21.15.32:0/4197435718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d0005450 con 0x55dbabb6ff40 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9c5ffb640 1 -- 172.21.15.32:0/4197435718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0035040 con 0x55dbabb6ff40 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9e91b3740 1 -- 172.21.15.32:0/4197435718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 msgr2=0x55dbabb79c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009930 tx=0x7fd9d0032fc0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9e91b3740 1 -- 172.21.15.32:0/4197435718 shutdown_connections 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb7a800 0x55dbabb92490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.571+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb79c50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/4197435718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb6f620 0x55dbabb86290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- 172.21.15.32:0/4197435718 >> 172.21.15.32:0/4197435718 conn(0x55dbabb6bb30 msgr2=0x55dbabb729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- 172.21.15.32:0/4197435718 shutdown_connections 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- 172.21.15.32:0/4197435718 wait complete. 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- start start 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 Processor -- start 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb82b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01a9ee0 con 0x55dbabb6ff40 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01ab350 con 0x55dbabb6f620 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01aa5b0 con 0x55dbabb7a800 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9e91b3740 10 client.0 _ll_register_callbacks cb 0x7fd9d400bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.572+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50998/0 (socket says 172.21.15.32:50998) 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 -- 172.21.15.32:0/1724629894 learned_addr learned my addr 172.21.15.32:0/1724629894 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb82b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 -- 172.21.15.32:0/1724629894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 msgr2=0x55dbabb7e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 -- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 msgr2=0x55dbabb82b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb82b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 -- 172.21.15.32:0/1724629894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbabb7ec00 con 0x55dbabb6ff40 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb82b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9dffff640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0031440 tx=0x7fd9d0031e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bd1e3b0f2659c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0035070 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd9d00314a0 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9d0005450 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd9d0059040 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9e91b3740 10 client.16113 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.573+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dbabbf0c90 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d0005450 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fd9d00535c0 con 0x55dbabb6ff40 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 10 client.16113 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 20 client.16113 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabbf0eb0 0x55dbabbf32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 4 client.16113 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbabadc200 con 0x55dbabbf0eb0 2024-04-23T18:20:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabbf0eb0 0x55dbabbf32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9df7fe640 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabbf0eb0 0x55dbabbf32a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb7550efb86ac1d server_cookie=2bc3240801d93280 in_seq=0 out_seq=0 2024-04-23T18:20:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.574+0000 7fd9bf7fe640 10 client.16113 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.591+0000 7fd9bf7fe640 1 -- 172.21.15.32:0/1724629894 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbabadc200 con 0x55dbabbf0eb0 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.591+0000 7fd9bf7fe640 10 client.16113 handle_command_reply: tid=0 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 client.16113 shutdown 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 msgr2=0x55dbabb7dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0031440 tx=0x7fd9d0031e70 comp rx=0 tx=0).stop 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 shutdown_connections 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbabbf0eb0 0x55dbabbf32a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbabb7a800 0x55dbabb7e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbabb6ff40 0x55dbabb7dee0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9bf7fe640 0 client.16113 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 --2- 172.21.15.32:0/1724629894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbabb6f620 0x55dbabb82b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 >> 172.21.15.32:0/1724629894 conn(0x55dbabb6bb30 msgr2=0x55dbabb71020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 shutdown_connections 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 1 -- 172.21.15.32:0/1724629894 wait complete. 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.592+0000 7fd9e91b3740 20 client.16113 trim_cache size 0 max 16384 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:20:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:20:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:20:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 msgr2=0x7fd9c80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 0x7fd9c80a61d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80ce360 tx=0x7fd9d8087630 comp rx=0 tx=0).stop 2024-04-23T18:20:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 msgr2=0x7fd9e01a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800cce0 tx=0x7fd9d800a620 comp rx=0 tx=0).stop 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 shutdown_connections 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd9c80a3d10 0x7fd9c80a61d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e0199ff0 0x7fd9e01a45a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9e0106220 0x7fd9e0199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 --2- 172.21.15.32:0/2200181307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e0101350 0x7fd9e0199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 >> 172.21.15.32:0/2200181307 conn(0x7fd9e0077d10 msgr2=0x7fd9e00ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 shutdown_connections 2024-04-23T18:20:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:14.593+0000 7fd9e63bf640 1 -- 172.21.15.32:0/2200181307 wait complete. 2024-04-23T18:20:15.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:14 smithi032 ceph-mon[29221]: pgmap v157: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:15.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:14 smithi040 ceph-mon[32626]: pgmap v157: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:15.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:14 smithi064 ceph-mon[33062]: pgmap v157: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 41 MiB/s wr, 1.94k op/s 2024-04-23T18:20:17.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:16 smithi032 ceph-mon[29221]: pgmap v158: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 22 MiB/s wr, 699 op/s 2024-04-23T18:20:17.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:17.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:16 smithi040 ceph-mon[32626]: pgmap v158: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 22 MiB/s wr, 699 op/s 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 -- 172.21.15.32:0/1798443822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b40742f0 msgr2=0x7f72b40746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/1798443822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b40742f0 0x7f72b40746d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009900 tx=0x7f72b002a480 comp rx=0 tx=0).stop 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 -- 172.21.15.32:0/1798443822 shutdown_connections 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/1798443822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b4112f40 0x7f72b4115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/1798443822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 0x7f72b4112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/1798443822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b40742f0 0x7f72b40746d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 -- 172.21.15.32:0/1798443822 >> 172.21.15.32:0/1798443822 conn(0x7f72b40fda40 msgr2=0x7f72b40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 -- 172.21.15.32:0/1798443822 shutdown_connections 2024-04-23T18:20:17.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.133+0000 7f72bb7cd640 1 -- 172.21.15.32:0/1798443822 wait complete. 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 Processor -- start 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 -- start start 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b40742f0 0x7f72b419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 0x7f72b419f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41187e0 con 0x7f72b4112f40 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4118660 con 0x7f72b40742f0 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72bb7cd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4118960 con 0x7f72b4074c10 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72b9d6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72b9d6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44976/0 (socket says 172.21.15.32:44976) 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.134+0000 7f72b9d6a640 1 -- 172.21.15.32:0/2658363900 learned_addr learned my addr 172.21.15.32:0/2658363900 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b8d68640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 0x7f72b419f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9569640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b40742f0 0x7f72b419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 -- 172.21.15.32:0/2658363900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 msgr2=0x7f72b419f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 0x7f72b419f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 -- 172.21.15.32:0/2658363900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b40742f0 msgr2=0x7f72b419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b40742f0 0x7f72b419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 -- 172.21.15.32:0/2658363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b41a3db0 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72b9d6a640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00cab0 tx=0x7f72ac00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00ae20 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b41a40a0 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72ac00f900 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac018a50 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.135+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b41ab930 con 0x7f72b4112f40 2024-04-23T18:20:17.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.136+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b4069250 con 0x7f72b4112f40 2024-04-23T18:20:17.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.137+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f72ac00f450 con 0x7f72b4112f40 2024-04-23T18:20:17.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.138+0000 7f72aa7fc640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 0x7f72a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.138+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f72ac0d0fa0 con 0x7f72b4112f40 2024-04-23T18:20:17.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.138+0000 7f72b9569640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 0x7f72a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.139+0000 7f72b9569640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 0x7f72a00a61e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72b0031440 tx=0x7f72b00095f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.143+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72ac014030 con 0x7f72b4112f40 2024-04-23T18:20:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:16 smithi064 ceph-mon[33062]: pgmap v158: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 22 MiB/s wr, 699 op/s 2024-04-23T18:20:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.368+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72b402d5c0 con 0x7f72b4112f40 2024-04-23T18:20:17.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.369+0000 7f72aa7fc640 1 -- 172.21.15.32:0/2658363900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11852 (secure 0 0 0) 0x7f72ac09d690 con 0x7f72b4112f40 2024-04-23T18:20:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:20:17.363 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:19:50.863737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:20:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-23T18:20:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 msgr2=0x7f72a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 0x7f72a00a61e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72b0031440 tx=0x7f72b00095f0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 msgr2=0x7f72b41a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00cab0 tx=0x7f72ac00cf70 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 shutdown_connections 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72a00a3d20 0x7f72a00a61e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b4112f40 0x7f72b41a3680 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72b4074c10 0x7f72b419f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 --2- 172.21.15.32:0/2658363900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72b40742f0 0x7f72b419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 >> 172.21.15.32:0/2658363900 conn(0x7f72b40fda40 msgr2=0x7f72b40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 shutdown_connections 2024-04-23T18:20:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.371+0000 7f72bb7cd640 1 -- 172.21.15.32:0/2658363900 wait complete. 2024-04-23T18:20:17.375 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 8224a456 --release --await' 2024-04-23T18:20:17.376 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 8224a456 --release --await 2024-04-23T18:20:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- 172.21.15.32:0/3457230182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0103400 msgr2=0x7f99e010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/3457230182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0103400 0x7f99e010de00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009cd0 tx=0x7f99d802a8e0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- 172.21.15.32:0/3457230182 shutdown_connections 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/3457230182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0103400 0x7f99e010de00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/3457230182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/3457230182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0101180 0x7f99e0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- 172.21.15.32:0/3457230182 >> 172.21.15.32:0/3457230182 conn(0x7f99e0077f70 msgr2=0x7f99e0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- 172.21.15.32:0/3457230182 shutdown_connections 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- 172.21.15.32:0/3457230182 wait complete. 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 Processor -- start 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.492+0000 7f99e6fe8640 1 -- start start 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0103400 0x7f99e019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01111a0 con 0x7f99e0101180 2024-04-23T18:20:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99e0111020 con 0x7f99e0103400 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99e0111320 con 0x7f99e0102a60 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44992/0 (socket says 172.21.15.32:44992) 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 -- 172.21.15.32:0/366365964 learned_addr learned my addr 172.21.15.32:0/366365964 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e5585640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0103400 0x7f99e019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99dffff640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 -- 172.21.15.32:0/366365964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 msgr2=0x7f99e0197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 -- 172.21.15.32:0/366365964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0103400 msgr2=0x7f99e019bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0103400 0x7f99e019bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 -- 172.21.15.32:0/366365964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e019c6f0 con 0x7f99e0101180 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99dffff640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e4d84640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f99d000d950 tx=0x7f99d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99ddffb640 1 -- 172.21.15.32:0/366365964 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0017030 con 0x7f99e0101180 2024-04-23T18:20:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99ddffb640 1 -- 172.21.15.32:0/366365964 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d0004020 con 0x7f99e0101180 2024-04-23T18:20:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e019c9e0 con 0x7f99e0101180 2024-04-23T18:20:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.493+0000 7f99ddffb640 1 -- 172.21.15.32:0/366365964 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00047a0 con 0x7f99e0101180 2024-04-23T18:20:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.494+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e01a4270 con 0x7f99e0101180 2024-04-23T18:20:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99ddffb640 1 -- 172.21.15.32:0/366365964 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f99d0009a30 con 0x7f99e0101180 2024-04-23T18:20:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 Processor -- start 2024-04-23T18:20:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 -- start start 2024-04-23T18:20:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53647f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c53648480 0x558c5364a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c535a26d0 con 0x558c5363d490 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558c5353f1a0 con 0x558c53648480 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.496+0000 7f99e9dce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c5359e690 con 0x558c5363ddb0 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44998/0 (socket says 172.21.15.32:44998) 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 -- 172.21.15.32:0/2723548861 learned_addr learned my addr 172.21.15.32:0/2723548861 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99ddffb640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 0x7f99c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e4d84640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53647f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 0x7f99c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99dffff640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c53648480 0x558c5364a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 -- 172.21.15.32:0/2723548861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 msgr2=0x558c53647f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53647f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 -- 172.21.15.32:0/2723548861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c53648480 msgr2=0x558c5364a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c53648480 0x558c5364a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 -- 172.21.15.32:0/2723548861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c535495b0 con 0x558c5363d490 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99ddffb640 1 -- 172.21.15.32:0/366365964 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f99d00d02c0 con 0x7f99e0101180 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e4d84640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53647f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99e5585640 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004af0 tx=0x7f99d80318d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4692ba517bc2df0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99c6ffd640 1 -- 172.21.15.32:0/2723548861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8031be0 con 0x558c5363d490 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99c6ffd640 1 -- 172.21.15.32:0/2723548861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d8050070 con 0x558c5363d490 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.497+0000 7f99c6ffd640 1 -- 172.21.15.32:0/2723548861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8030c90 con 0x558c5363d490 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- 172.21.15.32:0/2723548861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 msgr2=0x558c5363d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004af0 tx=0x7f99d80318d0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e5585640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 0x7f99c80a61e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99d804f000 tx=0x7f99d804d000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- 172.21.15.32:0/2723548861 shutdown_connections 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c53648480 0x558c5364a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53647f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2723548861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363d490 0x558c5363d870 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- 172.21.15.32:0/2723548861 >> 172.21.15.32:0/2723548861 conn(0x558c536397b0 msgr2=0x558c53639b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- 172.21.15.32:0/2723548861 shutdown_connections 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- 172.21.15.32:0/2723548861 wait complete. 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 -- start start 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.498+0000 7f99e9dce740 1 Processor -- start 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53647e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53645180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c5364b310 con 0x558c53648480 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558c5353cb20 con 0x558c5363d490 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c5364b490 con 0x558c5363ddb0 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e9dce740 10 client.0 _ll_register_callbacks cb 0x558c5353c530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45010/0 (socket says 172.21.15.32:45010) 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 -- 172.21.15.32:0/2782582474 learned_addr learned my addr 172.21.15.32:0/2782582474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e4d84640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53647e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99dffff640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53645180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 -- 172.21.15.32:0/2782582474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 msgr2=0x558c53645180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53645180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 -- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 msgr2=0x558c53647e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53647e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99e5585640 1 -- 172.21.15.32:0/2782582474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c532b4920 con 0x558c53648480 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.499+0000 7f99dffff640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53645180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e5585640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f99d802a920 tx=0x7f99d8005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bc2b8c26add61d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8062060 con 0x558c53648480 2024-04-23T18:20:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99d8050070 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f99d80305a0 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f99d8004b40 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 10 client.16131 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558c53380d30 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d8050070 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f99d8030070 con 0x558c53648480 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 10 client.16131 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 10 client.16131 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 20 client.16131 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c5365a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 4 client.16131 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.500+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c53560500 con 0x558c536584f0 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.501+0000 7f99e4d84640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c5365a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.501+0000 7f99bffff640 10 client.16131 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.501+0000 7f99e4d84640 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c5365a8e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4c28e932f362c47 server_cookie=b81fbeb877c854e3 in_seq=0 out_seq=0 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99bffff640 1 -- 172.21.15.32:0/2782582474 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c53560500 con 0x558c536584f0 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99bffff640 10 client.16131 handle_command_reply: tid=0 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 client.16131 shutdown 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 msgr2=0x558c53645790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f99d802a920 tx=0x7f99d8005c30 comp rx=0 tx=0).stop 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 shutdown_connections 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c5365a8e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99bffff640 0 client.16131 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c53648480 0x558c53645790 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c5363ddb0 0x558c53645180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 --2- 172.21.15.32:0/2782582474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53647e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 >> 172.21.15.32:0/2782582474 conn(0x558c536397b0 msgr2=0x558c536389d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 shutdown_connections 2024-04-23T18:20:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 1 -- 172.21.15.32:0/2782582474 wait complete. 2024-04-23T18:20:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.502+0000 7f99e9dce740 20 client.16131 trim_cache size 0 max 16384 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 Processor -- start 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 -- start start 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53654200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c5365ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01ae200 con 0x558c5363ddb0 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01abaa0 con 0x558c5363d490 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e9dce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01a98f0 con 0x558c53648480 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45020/0 (socket says 172.21.15.32:45020) 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 -- 172.21.15.32:0/526067884 learned_addr learned my addr 172.21.15.32:0/526067884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e5585640 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c5365ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99e4d84640 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53654200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 -- 172.21.15.32:0/526067884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 msgr2=0x558c5365ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c5365ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 -- 172.21.15.32:0/526067884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 msgr2=0x558c53654200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53654200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.520+0000 7f99dffff640 1 -- 172.21.15.32:0/526067884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c5363caa0 con 0x558c5363ddb0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99dffff640 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009900 tx=0x7f99d4009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29061fadfe235d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99c67fc640 1 -- 172.21.15.32:0/526067884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4013ce0 con 0x558c5363ddb0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99c67fc640 1 -- 172.21.15.32:0/526067884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d40035f0 con 0x558c5363ddb0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99c67fc640 1 -- 172.21.15.32:0/526067884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d401c640 con 0x558c5363ddb0 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 -- 172.21.15.32:0/526067884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 msgr2=0x558c536479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009900 tx=0x7f99d4009dc0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 -- 172.21.15.32:0/526067884 shutdown_connections 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c5365ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c536479c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 --2- 172.21.15.32:0/526067884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53654200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 -- 172.21.15.32:0/526067884 >> 172.21.15.32:0/526067884 conn(0x558c536397b0 msgr2=0x558c5363f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 -- 172.21.15.32:0/526067884 shutdown_connections 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.521+0000 7f99e9dce740 1 -- 172.21.15.32:0/526067884 wait complete. 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 -- start start 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 Processor -- start 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53650710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c53641430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01a9f60 con 0x558c5363ddb0 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01ab3d0 con 0x558c5363d490 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01aa630 con 0x558c53648480 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e9dce740 10 client.0 _ll_register_callbacks cb 0x7f99d8030d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99dffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45024/0 (socket says 172.21.15.32:45024) 2024-04-23T18:20:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99dffff640 1 -- 172.21.15.32:0/1258660162 learned_addr learned my addr 172.21.15.32:0/1258660162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.522+0000 7f99e5585640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c53641430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e4d84640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53650710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 -- 172.21.15.32:0/1258660162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 msgr2=0x558c53641430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c53641430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 -- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 msgr2=0x558c53650710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53650710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 -- 172.21.15.32:0/1258660162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c53641c60 con 0x558c5363ddb0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e5585640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c53641430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e4d84640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53650710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99dffff640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99d40137d0 tx=0x7f99d40039b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b791d3acb32eca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d402c060 con 0x558c5363ddb0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f99d4004050 con 0x558c5363ddb0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99d40035f0 con 0x558c5363ddb0 2024-04-23T18:20:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f99d4004270 con 0x558c5363ddb0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e9dce740 10 client.16137 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558c53642fa0 con 0x558c5363ddb0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d40035f0 con 0x558c5363ddb0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.523+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f99d401a020 con 0x558c5363ddb0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 10 client.16137 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 10 client.16137 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 20 client.16137 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c536c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 4 client.16137 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8224a456", "release": true, "await": true}] 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8224a456", "release": true, "await": true}) -- 0x558c535b00f0 con 0x558c536584f0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e4d84640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c536c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99bffff640 10 client.16137 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.524+0000 7f99e4d84640 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c536c1b00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c9703482a557bd9 server_cookie=7aeed8d8c2038012 in_seq=0 out_seq=0 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99bffff640 1 -- 172.21.15.32:0/1258660162 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558c535b00f0 con 0x558c536584f0 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99bffff640 10 client.16137 handle_command_reply: tid=0 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 client.16137 shutdown 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 msgr2=0x558c53650c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99d40137d0 tx=0x7f99d40039b0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 shutdown_connections 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558c536584f0 0x558c536c1b00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c53648480 0x558c53641430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c5363ddb0 0x558c53650c50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99bffff640 0 client.16137 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 --2- 172.21.15.32:0/1258660162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c5363d490 0x558c53650710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 >> 172.21.15.32:0/1258660162 conn(0x558c536397b0 msgr2=0x558c536531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 shutdown_connections 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 1 -- 172.21.15.32:0/1258660162 wait complete. 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e9dce740 20 client.16137 trim_cache size 0 max 16384 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 33, 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 16, 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:20:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "8224a456": { 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 16, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 70.0, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:20:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:17.528 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 msgr2=0x7f99c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.531+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 0x7f99c80a61e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99d804f000 tx=0x7f99d804d000 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 msgr2=0x7f99e0197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f99d000d950 tx=0x7f99d000de10 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 shutdown_connections 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e0103400 0x7f99e019bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99e0102a60 0x7f99e0197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f99c80a3d20 0x7f99c80a61e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 --2- 172.21.15.32:0/366365964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e0101180 0x7f99e0197700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 >> 172.21.15.32:0/366365964 conn(0x7f99e0077f70 msgr2=0x7f99e0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 shutdown_connections 2024-04-23T18:20:17.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:17.532+0000 7f99e6fe8640 1 -- 172.21.15.32:0/366365964 wait complete. 2024-04-23T18:20:17.537 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8224a456', seconds elapsed: 11.6 2024-04-23T18:20:18.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:17 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2658363900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:17 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2658363900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:17 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2658363900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 290.0 s, 766.4 tps, lat 314.119 ms stddev 1713.783 2024-04-23T18:20:19.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:18 smithi032 ceph-mon[29221]: pgmap v159: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 451 op/s 2024-04-23T18:20:19.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:18 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:18 smithi040 ceph-mon[32626]: pgmap v159: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 451 op/s 2024-04-23T18:20:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:18 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:18 smithi064 ceph-mon[33062]: pgmap v159: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 451 op/s 2024-04-23T18:20:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:18 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:20.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:19 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:20.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:19 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:20.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:19 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:20:20.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:19 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:20.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:19 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:20.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:19 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:20:20.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:19 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:20.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:19 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:20.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:19 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:20:21.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:20 smithi040 ceph-mon[32626]: pgmap v160: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 20 MiB/s wr, 516 op/s 2024-04-23T18:20:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:20 smithi064 ceph-mon[33062]: pgmap v160: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 20 MiB/s wr, 516 op/s 2024-04-23T18:20:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:20 smithi032 ceph-mon[29221]: pgmap v160: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 20 MiB/s wr, 516 op/s 2024-04-23T18:20:23.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:22 smithi040 ceph-mon[32626]: pgmap v161: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.1 MiB/s wr, 70 op/s 2024-04-23T18:20:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:22 smithi064 ceph-mon[33062]: pgmap v161: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.1 MiB/s wr, 70 op/s 2024-04-23T18:20:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:22 smithi032 ceph-mon[29221]: pgmap v161: 97 pgs: 97 active+clean; 3.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.1 MiB/s wr, 70 op/s 2024-04-23T18:20:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 295.0 s, 3987.8 tps, lat 25.173 ms stddev 30.466 2024-04-23T18:20:25.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:24 smithi040 ceph-mon[32626]: pgmap v162: 97 pgs: 97 active+clean; 3.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-04-23T18:20:25.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:24 smithi064 ceph-mon[33062]: pgmap v162: 97 pgs: 97 active+clean; 3.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-04-23T18:20:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:24 smithi032 ceph-mon[29221]: pgmap v162: 97 pgs: 97 active+clean; 3.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-04-23T18:20:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:26 smithi040 ceph-mon[32626]: pgmap v163: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:27.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:26 smithi064 ceph-mon[33062]: pgmap v163: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:27.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:26 smithi032 ceph-mon[29221]: pgmap v163: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.734+0000 7faa0e1bb640 1 -- 172.21.15.32:0/2407349728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa08101660 msgr2=0x7faa0810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.734+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/2407349728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa08101660 0x7faa0810e3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009900 tx=0x7fa9fc02a410 comp rx=0 tx=0).stop 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 -- 172.21.15.32:0/2407349728 shutdown_connections 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/2407349728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0810e8f0 0x7faa08110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/2407349728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa08101660 0x7faa0810e3b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/2407349728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 0x7faa08101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 -- 172.21.15.32:0/2407349728 >> 172.21.15.32:0/2407349728 conn(0x7faa08077f70 msgr2=0x7faa08078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 -- 172.21.15.32:0/2407349728 shutdown_connections 2024-04-23T18:20:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.735+0000 7faa0e1bb640 1 -- 172.21.15.32:0/2407349728 wait complete. 2024-04-23T18:20:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.736+0000 7faa0e1bb640 1 Processor -- start 2024-04-23T18:20:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.736+0000 7faa0e1bb640 1 -- start start 2024-04-23T18:20:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 0x7faa081a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 0x7faa081a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa08114440 con 0x7faa08101660 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa081142c0 con 0x7faa08100d40 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa0e1bb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa081145c0 con 0x7faa0810e8f0 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa07fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa07fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49116/0 (socket says 172.21.15.32:49116) 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.737+0000 7faa07fff640 1 -- 172.21.15.32:0/1132767071 learned_addr learned my addr 172.21.15.32:0/1132767071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa077fe640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 0x7faa081a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 0x7faa081a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa07fff640 1 -- 172.21.15.32:0/1132767071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 msgr2=0x7faa081a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa07fff640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 0x7faa081a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa07fff640 1 -- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 msgr2=0x7faa081a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa07fff640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 0x7faa081a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa07fff640 1 -- 172.21.15.32:0/1132767071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0819afc0 con 0x7faa0810e8f0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.738+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 0x7faa081a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa07fff640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa0000eb10 tx=0x7faa0000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa04ff9640 1 -- 172.21.15.32:0/1132767071 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0000cdc0 con 0x7faa0810e8f0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa04ff9640 1 -- 172.21.15.32:0/1132767071 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa00005020 con 0x7faa0810e8f0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0819b2b0 con 0x7faa0810e8f0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa04ff9640 1 -- 172.21.15.32:0/1132767071 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00010690 con 0x7faa0810e8f0 2024-04-23T18:20:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.739+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa081a75c0 con 0x7faa0810e8f0 2024-04-23T18:20:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.742+0000 7faa04ff9640 1 -- 172.21.15.32:0/1132767071 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7faa000054d0 con 0x7faa0810e8f0 2024-04-23T18:20:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 Processor -- start 2024-04-23T18:20:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 -- start start 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480f4f40 0x55b8480ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480ff800 0x55b848101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b848059970 con 0x55b8480f4f40 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b847ff64a0 con 0x55b8480ff800 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.743+0000 7faa10fae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b848055930 con 0x55b8480f4620 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49130/0 (socket says 172.21.15.32:49130) 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 -- 172.21.15.32:0/1882373314 learned_addr learned my addr 172.21.15.32:0/1882373314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480ff800 0x55b848101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa04ff9640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 0x7fa9f00a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 -- 172.21.15.32:0/1882373314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480ff800 msgr2=0x55b848101bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480ff800 0x55b848101bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 -- 172.21.15.32:0/1882373314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480f4f40 msgr2=0x55b8480ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480f4f40 0x55b8480ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 -- 172.21.15.32:0/1882373314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b847d6c210 con 0x55b8480f4620 2024-04-23T18:20:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa07fff640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 0x7fa9f00a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.744+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0057c0 tx=0x7fa9fc005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12fc6654a554b10 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7faa04ff9640 1 -- 172.21.15.32:0/1132767071 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7faa000d0ef0 con 0x7faa0810e8f0 2024-04-23T18:20:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7fa9edffb640 1 -- 172.21.15.32:0/1882373314 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc035040 con 0x55b8480f4620 2024-04-23T18:20:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7fa9edffb640 1 -- 172.21.15.32:0/1882373314 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9fc0038c0 con 0x55b8480f4620 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7fa9edffb640 1 -- 172.21.15.32:0/1882373314 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc004920 con 0x55b8480f4620 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7faa10fae740 1 -- 172.21.15.32:0/1882373314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 msgr2=0x55b8480f4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0057c0 tx=0x7fa9fc005730 comp rx=0 tx=0).stop 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.745+0000 7faa07fff640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 0x7fa9f00a62d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa0000c760 tx=0x7faa0001c000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 -- 172.21.15.32:0/1882373314 shutdown_connections 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480ff800 0x55b848101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480f4f40 0x55b8480ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 --2- 172.21.15.32:0/1882373314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480f4a00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.746+0000 7faa10fae740 1 -- 172.21.15.32:0/1882373314 >> 172.21.15.32:0/1882373314 conn(0x55b8480f0b30 msgr2=0x55b8480f0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.747+0000 7faa10fae740 1 -- 172.21.15.32:0/1882373314 shutdown_connections 2024-04-23T18:20:27.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.747+0000 7faa10fae740 1 -- 172.21.15.32:0/1882373314 wait complete. 2024-04-23T18:20:27.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.747+0000 7faa10fae740 1 -- start start 2024-04-23T18:20:27.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.747+0000 7faa10fae740 1 Processor -- start 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480ff210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 0x55b8480fc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b8480fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b8481023d0 con 0x55b8480ff800 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b847ff40b0 con 0x55b8480f4f40 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b848102550 con 0x55b8480f4620 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa10fae740 10 client.0 _ll_register_callbacks cb 0x55b847fe1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.748+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480ff210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480ff210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49136/0 (socket says 172.21.15.32:49136) 2024-04-23T18:20:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa06ffd640 1 -- 172.21.15.32:0/1943669858 learned_addr learned my addr 172.21.15.32:0/1943669858 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 0x55b8480fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b8480fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 -- 172.21.15.32:0/1943669858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 msgr2=0x55b8480ff210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480ff210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 -- 172.21.15.32:0/1943669858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 msgr2=0x55b8480fcb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b8480fcb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 -- 172.21.15.32:0/1943669858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b847e38620 con 0x55b8480f4f40 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.749+0000 7faa077fe640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 0x55b8480fc500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b8480f5bc0 tx=0x7fa9f800fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15f3fcfeb2fdf6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7fa9e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f801d060 con 0x55b8480f4f40 2024-04-23T18:20:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9fc0038c0 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa9f800daa0 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa9f800dce0 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.750+0000 7faa10fae740 10 client.25763 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b84810e440 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f800de80 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa9f80105a0 con 0x55b8480f4f40 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 10 client.25763 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 10 client.25763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 20 client.25763 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b84810fda0 0x55b8481121c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 4 client.25763 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.751+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa9fc0038c0 con 0x55b84810fda0 2024-04-23T18:20:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.752+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b84810fda0 0x55b8481121c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.753+0000 7faa06ffd640 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b84810fda0 0x55b8481121c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c114697941e2be84 server_cookie=dff5562948059ab6 in_seq=0 out_seq=0 2024-04-23T18:20:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.753+0000 7fa9e6ffd640 10 client.25763 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/1943669858 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa9fc0038c0 con 0x55b84810fda0 2024-04-23T18:20:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7fa9e6ffd640 10 client.25763 handle_command_reply: tid=0 2024-04-23T18:20:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 client.25763 shutdown 2024-04-23T18:20:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 msgr2=0x55b8480fc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 0x55b8480fc500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b8480f5bc0 tx=0x7fa9f800fb20 comp rx=0 tx=0).stop 2024-04-23T18:20:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 shutdown_connections 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b84810fda0 0x55b8481121c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b8480fcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4f40 0x55b8480fc500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.754+0000 7faa10fae740 1 --2- 172.21.15.32:0/1943669858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4620 0x55b8480ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.755+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 >> 172.21.15.32:0/1943669858 conn(0x55b8480f0b30 msgr2=0x55b8480f68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.755+0000 7fa9e6ffd640 0 client.25763 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.755+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 shutdown_connections 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.755+0000 7faa10fae740 1 -- 172.21.15.32:0/1943669858 wait complete. 2024-04-23T18:20:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.755+0000 7faa10fae740 20 client.25763 trim_cache size 0 max 16384 2024-04-23T18:20:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 Processor -- start 2024-04-23T18:20:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 -- start start 2024-04-23T18:20:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848117ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa081b2590 con 0x55b8480ff800 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa081afe30 con 0x55b8480f4620 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa10fae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa081adc80 con 0x55b8480f4f40 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49152/0 (socket says 172.21.15.32:49152) 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:52516/0 (socket says 172.21.15.32:52516) 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 -- 172.21.15.32:0/3202670823 learned_addr learned my addr 172.21.15.32:0/3202670823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848117ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 -- 172.21.15.32:0/3202670823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 msgr2=0x55b84810b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 -- 172.21.15.32:0/3202670823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 msgr2=0x55b848117ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848117ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa077fe640 1 -- 172.21.15.32:0/3202670823 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b848112840 con 0x55b8480f4f40 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.783+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.784+0000 7faa077fe640 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800a350 tx=0x7fa9f8005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90dac6e4d3070d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.784+0000 7fa9ed7fa640 1 -- 172.21.15.32:0/3202670823 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f802f070 con 0x55b8480f4f40 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.784+0000 7fa9ed7fa640 1 -- 172.21.15.32:0/3202670823 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f800de80 con 0x55b8480f4f40 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.784+0000 7fa9ed7fa640 1 -- 172.21.15.32:0/3202670823 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8005500 con 0x55b8480f4f40 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- 172.21.15.32:0/3202670823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 msgr2=0x55b8480efb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800a350 tx=0x7fa9f8005b50 comp rx=0 tx=0).stop 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- 172.21.15.32:0/3202670823 shutdown_connections 2024-04-23T18:20:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848117ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b8480efb30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- 172.21.15.32:0/3202670823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b84810b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- 172.21.15.32:0/3202670823 >> 172.21.15.32:0/3202670823 conn(0x55b8480f0b30 msgr2=0x55b8480ef420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- 172.21.15.32:0/3202670823 shutdown_connections 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- 172.21.15.32:0/3202670823 wait complete. 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 -- start start 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 Processor -- start 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b848104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.785+0000 7faa10fae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa10fae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa081ae2f0 con 0x55b8480ff800 2024-04-23T18:20:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa10fae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa081af760 con 0x55b8480f4620 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa10fae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa081ae9c0 con 0x55b8480f4f40 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa10fae740 10 client.0 _ll_register_callbacks cb 0x7fa9f800abc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:52518/0 (socket says 172.21.15.32:52518) 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 -- 172.21.15.32:0/3787017413 learned_addr learned my addr 172.21.15.32:0/3787017413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa077fe640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b848104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 -- 172.21.15.32:0/3787017413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 msgr2=0x55b848104d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b848104d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 -- 172.21.15.32:0/3787017413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 msgr2=0x55b848105270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848105270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 -- 172.21.15.32:0/3787017413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8481058b0 con 0x55b8480f4620 2024-04-23T18:20:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa077fe640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b848104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa06ffd640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc02a9d0 tx=0x7fa9fc040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb6fad5033fbccc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7fa9e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc035040 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b848067390 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.786+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa9fc056450 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa9fc0043d0 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 10 client.25769 fetch_fsmap learned FSMap version 33 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b848176190 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b848067390 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa9fc03a040 con 0x55b8480f4620 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 10 client.25769 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:20:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 10 client.25769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 20 client.25769 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b848176420 0x55b848178810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 4 client.25769 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b848061200 con 0x55b848176420 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.787+0000 7faa077fe640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b848176420 0x55b848178810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.788+0000 7faa077fe640 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b848176420 0x55b848178810 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54be42a4db1b1038 server_cookie=ab68d3e0b0bbdd68 in_seq=0 out_seq=0 2024-04-23T18:20:27.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.788+0000 7fa9e6ffd640 10 client.25769 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7fa9e6ffd640 1 -- 172.21.15.32:0/3787017413 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b848061200 con 0x55b848176420 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7fa9e6ffd640 10 client.25769 handle_command_reply: tid=0 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 client.25769 shutdown 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 msgr2=0x55b8481079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc02a9d0 tx=0x7fa9fc040040 comp rx=0 tx=0).stop 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 shutdown_connections 2024-04-23T18:20:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b848176420 0x55b848178810 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8480ff800 0x55b848105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7fa9e6ffd640 0 client.25769 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b8480f4f40 0x55b848104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 --2- 172.21.15.32:0/3787017413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8480f4620 0x55b8481079a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 >> 172.21.15.32:0/3787017413 conn(0x55b8480f0b30 msgr2=0x55b8480efcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 shutdown_connections 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 1 -- 172.21.15.32:0/3787017413 wait complete. 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.798+0000 7faa10fae740 20 client.25769 trim_cache size 0 max 16384 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:20:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:20:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:20:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 msgr2=0x7fa9f00a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 0x7fa9f00a62d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa0000c760 tx=0x7faa0001c000 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 msgr2=0x7faa0819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa0000eb10 tx=0x7faa0000ef10 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 shutdown_connections 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa0810e8f0 0x7faa0819a700 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa9f00a3e10 0x7fa9f00a62d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa08101660 0x7faa081a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 --2- 172.21.15.32:0/1132767071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa08100d40 0x7faa081a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 >> 172.21.15.32:0/1132767071 conn(0x7faa08077f70 msgr2=0x7faa0810fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 shutdown_connections 2024-04-23T18:20:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:27.799+0000 7faa0e1bb640 1 -- 172.21.15.32:0/1132767071 wait complete. 2024-04-23T18:20:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 300.0 s, 4079.6 tps, lat 24.481 ms stddev 29.221 2024-04-23T18:20:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:28 smithi040 ceph-mon[32626]: pgmap v164: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:28 smithi064 ceph-mon[33062]: pgmap v164: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:28 smithi032 ceph-mon[29221]: pgmap v164: 97 pgs: 97 active+clean; 3.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 404 op/s 2024-04-23T18:20:31.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:30 smithi040 ceph-mon[32626]: pgmap v165: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 MiB/s wr, 745 op/s 2024-04-23T18:20:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:30 smithi064 ceph-mon[33062]: pgmap v165: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 MiB/s wr, 745 op/s 2024-04-23T18:20:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:30 smithi032 ceph-mon[29221]: pgmap v165: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 MiB/s wr, 745 op/s 2024-04-23T18:20:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:32 smithi040 ceph-mon[32626]: pgmap v166: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 35 MiB/s wr, 681 op/s 2024-04-23T18:20:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:32 smithi064 ceph-mon[33062]: pgmap v166: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 35 MiB/s wr, 681 op/s 2024-04-23T18:20:33.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:32 smithi032 ceph-mon[29221]: pgmap v166: 97 pgs: 97 active+clean; 3.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 35 MiB/s wr, 681 op/s 2024-04-23T18:20:33.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 305.0 s, 3067.8 tps, lat 32.661 ms stddev 61.790 2024-04-23T18:20:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:34 smithi040 ceph-mon[32626]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 MiB/s wr, 1.75k op/s 2024-04-23T18:20:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:34 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:34 smithi064 ceph-mon[33062]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 MiB/s wr, 1.75k op/s 2024-04-23T18:20:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:34 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:34 smithi032 ceph-mon[29221]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 MiB/s wr, 1.75k op/s 2024-04-23T18:20:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:34 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:36.109 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.6 2024-04-23T18:20:36.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.288+0000 7f6d0854f640 1 -- 172.21.15.32:0/2515041383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 msgr2=0x7f6d0011db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.288+0000 7f6d0854f640 1 --2- 172.21.15.32:0/2515041383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d0011db20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009ca0 tx=0x7f6cf802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.289+0000 7f6d0854f640 1 -- 172.21.15.32:0/2515041383 shutdown_connections 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.289+0000 7f6d0854f640 1 --2- 172.21.15.32:0/2515041383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d0011db20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.289+0000 7f6d0854f640 1 --2- 172.21.15.32:0/2515041383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 0x7f6d00109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.289+0000 7f6d0854f640 1 --2- 172.21.15.32:0/2515041383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 0x7f6d00108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.289+0000 7f6d0854f640 1 -- 172.21.15.32:0/2515041383 >> 172.21.15.32:0/2515041383 conn(0x7f6d0006e2e0 msgr2=0x7f6d0006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.290+0000 7f6d0854f640 1 -- 172.21.15.32:0/2515041383 shutdown_connections 2024-04-23T18:20:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.290+0000 7f6d0854f640 1 -- 172.21.15.32:0/2515041383 wait complete. 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 Processor -- start 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 -- start start 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 0x7f6d001a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 0x7f6d001a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00121070 con 0x7f6d00109490 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00120ef0 con 0x7f6d00110770 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d0854f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001211f0 con 0x7f6d0011b730 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.291+0000 7f6d06aec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49178/0 (socket says 172.21.15.32:49178) 2024-04-23T18:20:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 -- 172.21.15.32:0/528967558 learned_addr learned my addr 172.21.15.32:0/528967558 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d05aea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 0x7f6d001a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d062eb640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 0x7f6d001a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 -- 172.21.15.32:0/528967558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 msgr2=0x7f6d001a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 0x7f6d001a7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 -- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 msgr2=0x7f6d001a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 0x7f6d001a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.292+0000 7f6d06aec640 1 -- 172.21.15.32:0/528967558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d001ac560 con 0x7f6d0011b730 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.293+0000 7f6d06aec640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009bf0 tx=0x7f6cf80319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.293+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8003bf0 con 0x7f6d0011b730 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.293+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf8003d90 con 0x7f6d0011b730 2024-04-23T18:20:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.293+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d001ac7f0 con 0x7f6d0011b730 2024-04-23T18:20:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.294+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf802fab0 con 0x7f6d0011b730 2024-04-23T18:20:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.294+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d00074180 con 0x7f6d0011b730 2024-04-23T18:20:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.296+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0006f510 con 0x7f6d0011b730 2024-04-23T18:20:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.297+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6cf802fd30 con 0x7f6d0011b730 2024-04-23T18:20:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.297+0000 7f6cf77fe640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 0x7f6cec0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.298+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6cf8113d60 con 0x7f6d0011b730 2024-04-23T18:20:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.298+0000 7f6d062eb640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 0x7f6cec0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.298+0000 7f6d062eb640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 0x7f6cec0a6320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005fd0 tx=0x7f6cfc005850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.302+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6cf80e0590 con 0x7f6d0011b730 2024-04-23T18:20:36.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.670+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d001a88c0 con 0x7f6d0011b730 2024-04-23T18:20:36.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.672+0000 7f6cf77fe640 1 -- 172.21.15.32:0/528967558 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11853 (secure 0 0 0) 0x7f6cf80dfe90 con 0x7f6d0011b730 2024-04-23T18:20:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:20:36.666 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:20:33.800632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:20:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 msgr2=0x7f6cec0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 0x7f6cec0a6320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005fd0 tx=0x7f6cfc005850 comp rx=0 tx=0).stop 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 msgr2=0x7f6d001abe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009bf0 tx=0x7f6cf80319f0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 shutdown_connections 2024-04-23T18:20:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d0011b730 0x7f6d001abe30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d00110770 0x7f6d001a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6cec0a3e60 0x7f6cec0a6320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 --2- 172.21.15.32:0/528967558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d00109490 0x7f6d001a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.675+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 >> 172.21.15.32:0/528967558 conn(0x7f6d0006e2e0 msgr2=0x7f6d0010e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.676+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 shutdown_connections 2024-04-23T18:20:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.676+0000 7f6d0854f640 1 -- 172.21.15.32:0/528967558 wait complete. 2024-04-23T18:20:36.689 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 80.6 --await-for 42.0' 2024-04-23T18:20:36.689 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 80.6 --await-for 42.0 2024-04-23T18:20:36.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 -- 172.21.15.32:0/1728106895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd040103ae0 msgr2=0x7fd04010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1728106895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd040103ae0 0x7fd04010bfc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009900 tx=0x7fd02c02a480 comp rx=0 tx=0).stop 2024-04-23T18:20:36.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 -- 172.21.15.32:0/1728106895 shutdown_connections 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1728106895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04010c590 0x7fd04010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1728106895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd040103ae0 0x7fd04010bfc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1728106895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0401031c0 0x7fd0401035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.875+0000 7fd046e8a640 1 -- 172.21.15.32:0/1728106895 >> 172.21.15.32:0/1728106895 conn(0x7fd040077f50 msgr2=0x7fd040078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.876+0000 7fd046e8a640 1 -- 172.21.15.32:0/1728106895 shutdown_connections 2024-04-23T18:20:36.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.877+0000 7fd046e8a640 1 -- 172.21.15.32:0/1728106895 wait complete. 2024-04-23T18:20:36.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 Processor -- start 2024-04-23T18:20:36.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 -- start start 2024-04-23T18:20:36.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0401031c0 0x7fd0401a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 0x7fd04019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd040112220 con 0x7fd04010c590 2024-04-23T18:20:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401120a0 con 0x7fd0401031c0 2024-04-23T18:20:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd046e8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401123a0 con 0x7fd040103ae0 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35520/0 (socket says 172.21.15.32:35520) 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd044c26640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0401031c0 0x7fd0401a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.878+0000 7fd03ffff640 1 -- 172.21.15.32:0/1348362547 learned_addr learned my addr 172.21.15.32:0/1348362547 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd045427640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 0x7fd04019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 -- 172.21.15.32:0/1348362547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0401031c0 msgr2=0x7fd0401a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0401031c0 0x7fd0401a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 -- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 msgr2=0x7fd04019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 0x7fd04019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 -- 172.21.15.32:0/1348362547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04019b020 con 0x7fd040103ae0 2024-04-23T18:20:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd045427640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 0x7fd04019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.879+0000 7fd03ffff640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd02c005ab0 tx=0x7fd02c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.880+0000 7fd03dffb640 1 -- 172.21.15.32:0/1348362547 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c050070 con 0x7fd040103ae0 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.880+0000 7fd03dffb640 1 -- 172.21.15.32:0/1348362547 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd02c031b50 con 0x7fd040103ae0 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.880+0000 7fd03dffb640 1 -- 172.21.15.32:0/1348362547 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c035030 con 0x7fd040103ae0 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.880+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04019b2b0 con 0x7fd040103ae0 2024-04-23T18:20:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.880+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04006bb50 con 0x7fd040103ae0 2024-04-23T18:20:36.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.882+0000 7fd049ba2740 1 Processor -- start 2024-04-23T18:20:36.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 -- start start 2024-04-23T18:20:36.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d840360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d842c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56074d792370 con 0x56074d8361d0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56074d72f2e0 con 0x56074d8358b0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd049ba2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56074d78e330 con 0x56074d8408a0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd045427640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd045427640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:33700/0 (socket says 172.21.15.32:33700) 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.883+0000 7fd045427640 1 -- 172.21.15.32:0/1690547062 learned_addr learned my addr 172.21.15.32:0/1690547062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd03dffb640 1 -- 172.21.15.32:0/1348362547 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd02c053070 con 0x7fd040103ae0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd03ffff640 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d842c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd044c26640 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d840360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 -- 172.21.15.32:0/1690547062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 msgr2=0x56074d842c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d842c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 -- 172.21.15.32:0/1690547062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 msgr2=0x56074d840360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d840360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 -- 172.21.15.32:0/1690547062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56074d570d80 con 0x56074d8358b0 2024-04-23T18:20:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.884+0000 7fd045427640 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd038009ee0 tx=0x7fd03800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e81e54291dd90f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd03dffb640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 0x7fd0300a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd03dffb640 1 -- 172.21.15.32:0/1348362547 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd02c101f00 con 0x7fd040103ae0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd02affd640 1 -- 172.21.15.32:0/1690547062 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038011640 con 0x56074d8358b0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd044c26640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 0x7fd0300a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd02affd640 1 -- 172.21.15.32:0/1690547062 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd038011c90 con 0x56074d8358b0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.885+0000 7fd02affd640 1 -- 172.21.15.32:0/1690547062 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03801bcb0 con 0x56074d8358b0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 -- 172.21.15.32:0/1690547062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 msgr2=0x56074d835c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd038009ee0 tx=0x7fd03800f730 comp rx=0 tx=0).stop 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd044c26640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 0x7fd0300a62e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56074d836e70 tx=0x7fd034005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 -- 172.21.15.32:0/1690547062 shutdown_connections 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d842c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d840360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1690547062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d835c90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 -- 172.21.15.32:0/1690547062 >> 172.21.15.32:0/1690547062 conn(0x56074d831bd0 msgr2=0x56074d831fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 -- 172.21.15.32:0/1690547062 shutdown_connections 2024-04-23T18:20:36.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.886+0000 7fd049ba2740 1 -- 172.21.15.32:0/1690547062 wait complete. 2024-04-23T18:20:36.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.887+0000 7fd049ba2740 1 -- start start 2024-04-23T18:20:36.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.887+0000 7fd049ba2740 1 Processor -- start 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.887+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d83d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.887+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d83de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d83e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56074d843690 con 0x56074d8361d0 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56074d72cfb0 con 0x56074d8358b0 2024-04-23T18:20:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56074d843810 con 0x56074d8408a0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d83de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d83de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35212/0 (socket says 172.21.15.32:35212) 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd03ffff640 1 -- 172.21.15.32:0/1911050648 learned_addr learned my addr 172.21.15.32:0/1911050648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd044c26640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d83d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d83e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 10 client.0 _ll_register_callbacks cb 0x56074d72c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 -- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 msgr2=0x56074d83d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d83d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 -- 172.21.15.32:0/1911050648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 msgr2=0x56074d83de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d83de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd045427640 1 -- 172.21.15.32:0/1911050648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56074d5d1510 con 0x56074d8408a0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.888+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd038011c90 con 0x56074d8408a0 2024-04-23T18:20:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd044c26640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d83d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd045427640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d83e350 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd038006010 tx=0x7fd038012e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d4b6de7b10bed2d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd01ffff640 10 client.25762 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd01ffff640 1 -- 172.21.15.32:0/1911050648 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038004c40 con 0x56074d8408a0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd01ffff640 1 -- 172.21.15.32:0/1911050648 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd038004de0 con 0x56074d8408a0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd01ffff640 1 -- 172.21.15.32:0/1911050648 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd03801ba30 con 0x56074d8408a0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd049ba2740 10 client.25762 fetch_fsmap learned FSMap version 34 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.889+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({fsmap=34}) -- 0x56074d464670 con 0x56074d8408a0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd01ffff640 1 -- 172.21.15.32:0/1911050648 <== mon.1 v2:172.21.15.64:3300/0 4 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fd038024030 con 0x56074d8408a0 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 10 client.25762 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 10 client.25762 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 20 client.25762 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d82cce0 0x56074d82f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 4 client.25762 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.890+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56074d79fd90 con 0x56074d82cce0 2024-04-23T18:20:36.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.891+0000 7fd044c26640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d82cce0 0x56074d82f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.891+0000 7fd044c26640 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d82cce0 0x56074d82f0f0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703eb15286de34ba server_cookie=a41538ee4a46f44b in_seq=0 out_seq=0 2024-04-23T18:20:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.891+0000 7fd01ffff640 10 client.25762 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd01ffff640 1 -- 172.21.15.32:0/1911050648 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56074d79fd90 con 0x56074d82cce0 2024-04-23T18:20:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd01ffff640 10 client.25762 handle_command_reply: tid=0 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 client.25762 shutdown 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 msgr2=0x56074d83e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d83e350 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd038006010 tx=0x7fd038012e50 comp rx=0 tx=0).stop 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 shutdown_connections 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d82cce0 0x56074d82f0f0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8408a0 0x56074d83e350 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8361d0 0x56074d83de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 --2- 172.21.15.32:0/1911050648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8358b0 0x56074d83d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.892+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 >> 172.21.15.32:0/1911050648 conn(0x56074d831bd0 msgr2=0x56074d828600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.893+0000 7fd01ffff640 0 client.25762 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.894+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 shutdown_connections 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.894+0000 7fd049ba2740 1 -- 172.21.15.32:0/1911050648 wait complete. 2024-04-23T18:20:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.894+0000 7fd049ba2740 20 client.25762 trim_cache size 0 max 16384 2024-04-23T18:20:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.941+0000 7fd049ba2740 1 Processor -- start 2024-04-23T18:20:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.941+0000 7fd049ba2740 1 -- start start 2024-04-23T18:20:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.941+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.941+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8358b0 0x56074d83e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd044c26640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd044c26640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35548/0 (socket says 172.21.15.32:35548) 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd044c26640 1 -- 172.21.15.32:0/2318069236 learned_addr learned my addr 172.21.15.32:0/2318069236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd049ba2740 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d862ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400726c0 con 0x56074d8358b0 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd04006ff30 con 0x56074d8361d0 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd04006ddc0 con 0x56074d831bd0 2024-04-23T18:20:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.942+0000 7fd045427640 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d862ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 -- 172.21.15.32:0/2318069236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 msgr2=0x56074d862ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d862ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 -- 172.21.15.32:0/2318069236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8358b0 msgr2=0x56074d83e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8358b0 0x56074d83e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 -- 172.21.15.32:0/2318069236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56074d842c90 con 0x56074d831bd0 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd044c26640 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd034009f10 tx=0x7fd03401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43fec78507c007ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd02a7fc640 1 -- 172.21.15.32:0/2318069236 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03401dcd0 con 0x56074d831bd0 2024-04-23T18:20:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd02a7fc640 1 -- 172.21.15.32:0/2318069236 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0340054d0 con 0x56074d831bd0 2024-04-23T18:20:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.943+0000 7fd02a7fc640 1 -- 172.21.15.32:0/2318069236 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03402b050 con 0x56074d831bd0 2024-04-23T18:20:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 msgr2=0x56074d84c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd034009f10 tx=0x7fd03401b000 comp rx=0 tx=0).stop 2024-04-23T18:20:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 shutdown_connections 2024-04-23T18:20:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d862ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d8358b0 0x56074d83e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 --2- 172.21.15.32:0/2318069236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d831bd0 0x56074d84c550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.945+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 >> 172.21.15.32:0/2318069236 conn(0x56074d831610 msgr2=0x56074d83f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.946+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 shutdown_connections 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.946+0000 7fd049ba2740 1 -- 172.21.15.32:0/2318069236 wait complete. 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.946+0000 7fd049ba2740 1 -- start start 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.946+0000 7fd049ba2740 1 Processor -- start 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d831bd0 0x56074d844d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d8b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd04006e410 con 0x56074d831bd0 2024-04-23T18:20:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd04006f860 con 0x56074d8361d0 2024-04-23T18:20:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd04006eac0 con 0x56074d8358b0 2024-04-23T18:20:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd049ba2740 10 client.0 _ll_register_callbacks cb 0x7fd03801b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd03ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35552/0 (socket says 172.21.15.32:35552) 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd03ffff640 1 -- 172.21.15.32:0/3444767103 learned_addr learned my addr 172.21.15.32:0/3444767103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd045427640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d8b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.947+0000 7fd044c26640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d831bd0 0x56074d844d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 -- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 msgr2=0x56074d8b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d8b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 -- 172.21.15.32:0/3444767103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d831bd0 msgr2=0x56074d844d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d831bd0 0x56074d844d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 -- 172.21.15.32:0/3444767103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56074d8b89c0 con 0x56074d8358b0 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd03ffff640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0ff5b0 tx=0x7fd02c02a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=345cb7eb94eb6595 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd01ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.948+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c05a040 con 0x56074d8358b0 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0340054d0 con 0x56074d8358b0 2024-04-23T18:20:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd02c0ff8c0 con 0x56074d8358b0 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd02c1054c0 con 0x56074d8358b0 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd049ba2740 10 client.25768 fetch_fsmap learned FSMap version 34 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56074d8bfdb0 con 0x56074d8358b0 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.949+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0340054d0 con 0x56074d8358b0 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fd02c0ca500 con 0x56074d8358b0 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 10 client.25768 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 10 client.25768 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 20 client.25768 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d8408a0 0x56074d8c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 4 client.25768 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 80.6, "await_for": 42.0}] 2024-04-23T18:20:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.950+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 80.6, "await_for": 42.0}) -- 0x56074d799c00 con 0x56074d8408a0 2024-04-23T18:20:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.953+0000 7fd044c26640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d8408a0 0x56074d8c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.954+0000 7fd044c26640 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d8408a0 0x56074d8c26e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb9901fc0647bdb server_cookie=49393bbff41fd735 in_seq=0 out_seq=0 2024-04-23T18:20:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:36.954+0000 7fd01ffff640 10 client.25768 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:36 smithi032 ceph-mon[29221]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 MiB/s wr, 1.89k op/s 2024-04-23T18:20:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:36 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/528967558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:36 smithi040 ceph-mon[32626]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 MiB/s wr, 1.89k op/s 2024-04-23T18:20:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:36 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/528967558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd01ffff640 1 -- 172.21.15.32:0/3444767103 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56074d799c00 con 0x56074d8408a0 2024-04-23T18:20:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd01ffff640 10 client.25768 handle_command_reply: tid=0 2024-04-23T18:20:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 client.25768 shutdown 2024-04-23T18:20:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 msgr2=0x56074d845240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0ff5b0 tx=0x7fd02c02a480 comp rx=0 tx=0).stop 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 shutdown_connections 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56074d8408a0 0x56074d8c26e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56074d8361d0 0x56074d8b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd01ffff640 0 client.25768 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56074d8358b0 0x56074d845240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 --2- 172.21.15.32:0/3444767103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56074d831bd0 0x56074d844d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 >> 172.21.15.32:0/3444767103 conn(0x56074d831610 msgr2=0x56074d827d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 shutdown_connections 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 1 -- 172.21.15.32:0/3444767103 wait complete. 2024-04-23T18:20:37.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.175+0000 7fd049ba2740 20 client.25768 trim_cache size 0 max 16384 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 34, 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 18, 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "9de88f28": { 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 18, 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:37.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 80.6, 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:37.172 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:20:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 msgr2=0x7fd0300a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 0x7fd0300a62e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56074d836e70 tx=0x7fd034005f90 comp rx=0 tx=0).stop 2024-04-23T18:20:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 msgr2=0x7fd0401a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd02c005ab0 tx=0x7fd02c003b90 comp rx=0 tx=0).stop 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 shutdown_connections 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd0300a3e20 0x7fd0300a62e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04010c590 0x7fd04019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd040103ae0 0x7fd0401a0be0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 --2- 172.21.15.32:0/1348362547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0401031c0 0x7fd0401a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 >> 172.21.15.32:0/1348362547 conn(0x7fd040077f50 msgr2=0x7fd040108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 shutdown_connections 2024-04-23T18:20:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:37.176+0000 7fd046e8a640 1 -- 172.21.15.32:0/1348362547 wait complete. 2024-04-23T18:20:37.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:36 smithi064 ceph-mon[33062]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 MiB/s wr, 1.89k op/s 2024-04-23T18:20:37.174 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:36 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/528967558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:37.180 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9de88f28', quiesce took 1.1 seconds. Will release after: 20.6 2024-04-23T18:20:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 310.0 s, 1917.9 tps, lat 32.464 ms stddev 40.422 2024-04-23T18:20:39.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:38 smithi040 ceph-mon[32626]: pgmap v169: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 MiB/s wr, 1.81k op/s 2024-04-23T18:20:39.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:38 smithi064 ceph-mon[33062]: pgmap v169: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 MiB/s wr, 1.81k op/s 2024-04-23T18:20:39.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:38 smithi032 ceph-mon[29221]: pgmap v169: 97 pgs: 97 active+clean; 3.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 MiB/s wr, 1.81k op/s 2024-04-23T18:20:40.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:39 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:40.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:39 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:39 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 -- 172.21.15.32:0/733247601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 msgr2=0x7fb708101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 --2- 172.21.15.32:0/733247601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708101500 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb704009900 tx=0x7fb70402a410 comp rx=0 tx=0).stop 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 -- 172.21.15.32:0/733247601 shutdown_connections 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 --2- 172.21.15.32:0/733247601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 0x7fb70810dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 --2- 172.21.15.32:0/733247601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb708101ad0 0x7fb708105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 --2- 172.21.15.32:0/733247601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708101500 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.900+0000 7fb70ff76640 1 -- 172.21.15.32:0/733247601 >> 172.21.15.32:0/733247601 conn(0x7fb708077fa0 msgr2=0x7fb7080783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- 172.21.15.32:0/733247601 shutdown_connections 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- 172.21.15.32:0/733247601 wait complete. 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 Processor -- start 2024-04-23T18:20:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- start start 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 0x7fb7081030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7081035e0 0x7fb7081a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb708111150 con 0x7fb708101120 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb708110fd0 con 0x7fb7081061a0 2024-04-23T18:20:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.901+0000 7fb70ff76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081112d0 con 0x7fb7081035e0 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35266/0 (socket says 172.21.15.32:35266) 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 -- 172.21.15.32:0/4250881573 learned_addr learned my addr 172.21.15.32:0/4250881573 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70e513640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7081035e0 0x7fb7081a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70d511640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 0x7fb7081030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 -- 172.21.15.32:0/4250881573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7081035e0 msgr2=0x7fb7081a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7081035e0 0x7fb7081a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 -- 172.21.15.32:0/4250881573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 msgr2=0x7fb7081030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 0x7fb7081030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 -- 172.21.15.32:0/4250881573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7081a70a0 con 0x7fb708101120 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70dd12640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb704005ab0 tx=0x7fb704003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb6feffd640 1 -- 172.21.15.32:0/4250881573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70403f070 con 0x7fb708101120 2024-04-23T18:20:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb6feffd640 1 -- 172.21.15.32:0/4250881573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb704031d60 con 0x7fb708101120 2024-04-23T18:20:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7081a7330 con 0x7fb708101120 2024-04-23T18:20:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb6feffd640 1 -- 172.21.15.32:0/4250881573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704035030 con 0x7fb708101120 2024-04-23T18:20:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.902+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7081a7820 con 0x7fb708101120 2024-04-23T18:20:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.905+0000 7fb6feffd640 1 -- 172.21.15.32:0/4250881573 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb704042070 con 0x7fb708101120 2024-04-23T18:20:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.905+0000 7fb712d6c740 1 Processor -- start 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 -- start start 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647000f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000fd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564700057970 con 0x5647000fd800 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5646ffff44a0 con 0x5647000f2f40 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb712d6c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564700053930 con 0x5647000f2620 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35272/0 (socket says 172.21.15.32:35272) 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 -- 172.21.15.32:0/2222172494 learned_addr learned my addr 172.21.15.32:0/2222172494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70dd12640 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000fd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70e513640 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647000f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb6feffd640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 0x7fb6f40a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 -- 172.21.15.32:0/2222172494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 msgr2=0x5647000f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647000f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 -- 172.21.15.32:0/2222172494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 msgr2=0x5647000fd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.906+0000 7fb70d511640 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000fd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb70d511640 1 -- 172.21.15.32:0/2222172494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ffd6a210 con 0x5647000fd800 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb70e513640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 0x7fb6f40a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb70d511640 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009c60 tx=0x7fb6f8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c520089c5fadabc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb6feffd640 1 -- 172.21.15.32:0/4250881573 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb7040f0f30 con 0x7fb708101120 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb6f3fff640 1 -- 172.21.15.32:0/2222172494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8004230 con 0x5647000fd800 2024-04-23T18:20:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb6f3fff640 1 -- 172.21.15.32:0/2222172494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f80043d0 con 0x5647000fd800 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.907+0000 7fb6f3fff640 1 -- 172.21.15.32:0/2222172494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8019670 con 0x5647000fd800 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- 172.21.15.32:0/2222172494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 msgr2=0x5647000ffbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb70e513640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 0x7fb6f40a6200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb700005fd0 tx=0x7fb700004a40 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009c60 tx=0x7fb6f8011810 comp rx=0 tx=0).stop 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- 172.21.15.32:0/2222172494 shutdown_connections 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x5647000ffbf0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000fd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2222172494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647000f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- 172.21.15.32:0/2222172494 >> 172.21.15.32:0/2222172494 conn(0x5647000eeb30 msgr2=0x5647000eef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- 172.21.15.32:0/2222172494 shutdown_connections 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- 172.21.15.32:0/2222172494 wait complete. 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 -- start start 2024-04-23T18:20:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.908+0000 7fb712d6c740 1 Processor -- start 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2620 0x5647000fd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x5647000fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564700100430 con 0x5647000f2f40 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5646ffff20b0 con 0x5647000f2620 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5647001005b0 con 0x5647000fd800 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb712d6c740 10 client.0 _ll_register_callbacks cb 0x5646fffdfd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35280/0 (socket says 172.21.15.32:35280) 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 -- 172.21.15.32:0/3700835950 learned_addr learned my addr 172.21.15.32:0/3700835950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70e513640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x5647000fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70dd12640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2620 0x5647000fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 -- 172.21.15.32:0/3700835950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 msgr2=0x5647000fa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x5647000fa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 -- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2620 msgr2=0x5647000fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2620 0x5647000fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 -- 172.21.15.32:0/3700835950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ffe36620 con 0x5647000f2f40 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb70d511640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8003f20 tx=0x7fb6f80052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ed793b63afd021 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.909+0000 7fb6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8028020 con 0x5647000f2f40 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb6f80035f0 con 0x5647000f2f40 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6f80043d0 con 0x5647000f2f40 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb6f8033050 con 0x5647000f2f40 2024-04-23T18:20:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 10 client.16179 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56470010c360 con 0x5647000f2f40 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f80043d0 con 0x5647000f2f40 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fb6f8018070 con 0x5647000f2f40 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 10 client.16179 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 10 client.16179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 20 client.16179 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56470010d760 0x56470010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 4 client.16179 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.910+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564700065390 con 0x56470010d760 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.911+0000 7fb70dd12640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56470010d760 0x56470010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.911+0000 7fb6f17fa640 10 client.16179 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.911+0000 7fb70dd12640 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56470010d760 0x56470010fb50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3227a1a7c72e694f server_cookie=34abfef6bd30d1ba in_seq=0 out_seq=0 2024-04-23T18:20:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb6f17fa640 1 -- 172.21.15.32:0/3700835950 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564700065390 con 0x56470010d760 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb6f17fa640 10 client.16179 handle_command_reply: tid=0 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 client.16179 shutdown 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 msgr2=0x5647000fa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8003f20 tx=0x7fb6f80052f0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 shutdown_connections 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56470010d760 0x56470010fb50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x5647000fa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2f40 0x5647000fa440 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 --2- 172.21.15.32:0/3700835950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2620 0x5647000fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb6f17fa640 0 client.16179 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 >> 172.21.15.32:0/3700835950 conn(0x5647000eeb30 msgr2=0x5647000f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 shutdown_connections 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 1 -- 172.21.15.32:0/3700835950 wait complete. 2024-04-23T18:20:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.912+0000 7fb712d6c740 20 client.16179 trim_cache size 0 max 16384 2024-04-23T18:20:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 Processor -- start 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 -- start start 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647001092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000edb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081b2590 con 0x5647000fd800 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081afdd0 con 0x5647000f2f40 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb712d6c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081adc20 con 0x5647000f2620 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb70e513640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb70e513640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35284/0 (socket says 172.21.15.32:35284) 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.927+0000 7fb70e513640 1 -- 172.21.15.32:0/217915108 learned_addr learned my addr 172.21.15.32:0/217915108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70dd12640 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647001092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70d511640 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 -- 172.21.15.32:0/217915108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 msgr2=0x5647001092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647001092f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 -- 172.21.15.32:0/217915108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 msgr2=0x5647000edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000edb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 -- 172.21.15.32:0/217915108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647000f2d70 con 0x5647000fd800 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb70e513640 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7000138e0 tx=0x7fb700015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded892d03dbe96d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb6f37fe640 1 -- 172.21.15.32:0/217915108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70001f950 con 0x5647000fd800 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb6f37fe640 1 -- 172.21.15.32:0/217915108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb700017070 con 0x5647000fd800 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.928+0000 7fb6f37fe640 1 -- 172.21.15.32:0/217915108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70001cdd0 con 0x5647000fd800 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- 172.21.15.32:0/217915108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 msgr2=0x564700115370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7000138e0 tx=0x7fb700015000 comp rx=0 tx=0).stop 2024-04-23T18:20:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- 172.21.15.32:0/217915108 shutdown_connections 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000fd800 0x564700115370 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x5647000edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 --2- 172.21.15.32:0/217915108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000f2620 0x5647001092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- 172.21.15.32:0/217915108 >> 172.21.15.32:0/217915108 conn(0x5647000eeb30 msgr2=0x5647000ed420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- 172.21.15.32:0/217915108 shutdown_connections 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- 172.21.15.32:0/217915108 wait complete. 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 -- start start 2024-04-23T18:20:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.929+0000 7fb712d6c740 1 Processor -- start 2024-04-23T18:20:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x564700102e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x564700103390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081ae290 con 0x5647000f2620 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081af700 con 0x5647000f2f40 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081ae960 con 0x5647000fd800 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 10 client.0 _ll_register_callbacks cb 0x7fb6f8005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35296/0 (socket says 172.21.15.32:35296) 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 -- 172.21.15.32:0/2430409244 learned_addr learned my addr 172.21.15.32:0/2430409244 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70e513640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x564700103390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70d511640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x564700102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 -- 172.21.15.32:0/2430409244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 msgr2=0x564700103390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x564700103390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 -- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 msgr2=0x564700102e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x564700102e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 -- 172.21.15.32:0/2430409244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564700103a80 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70e513640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x564700103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70d511640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x564700102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb70dd12640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7040bcc70 tx=0x7fb7040f3380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20465ad454d161af server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704049040 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb7040b95e0 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.930+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb700017070 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb7040bc8d0 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 10 client.16185 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564700173b60 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb700017070 con 0x5647000f2620 2024-04-23T18:20:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fb7040ee8f0 con 0x5647000f2620 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 10 client.16185 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 10 client.16185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 20 client.16185 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564700174260 0x564700176680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 4 client.16185 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56470005f200 con 0x564700174260 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.931+0000 7fb70d511640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564700174260 0x564700176680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.932+0000 7fb70d511640 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564700174260 0x564700176680 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa93c7626dc1c622 server_cookie=3c2296bd15234c43 in_seq=0 out_seq=0 2024-04-23T18:20:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.932+0000 7fb6f17fa640 10 client.16185 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb6f17fa640 1 -- 172.21.15.32:0/2430409244 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56470005f200 con 0x564700174260 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb6f17fa640 10 client.16185 handle_command_reply: tid=0 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 client.16185 shutdown 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 msgr2=0x564700105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7040bcc70 tx=0x7fb7040f3380 comp rx=0 tx=0).stop 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 shutdown_connections 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x564700174260 0x564700176680 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647000fd800 0x564700103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647000f2f40 0x564700102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 --2- 172.21.15.32:0/2430409244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647000f2620 0x564700105ac0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb6f17fa640 0 client.16185 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 >> 172.21.15.32:0/2430409244 conn(0x5647000eeb30 msgr2=0x5647000edcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 shutdown_connections 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 1 -- 172.21.15.32:0/2430409244 wait complete. 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.953+0000 7fb712d6c740 20 client.16185 trim_cache size 0 max 16384 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:20:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 msgr2=0x7fb6f40a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 0x7fb6f40a6200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb700005fd0 tx=0x7fb700004a40 comp rx=0 tx=0).stop 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 msgr2=0x7fb708102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb704005ab0 tx=0x7fb704003c80 comp rx=0 tx=0).stop 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 shutdown_connections 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7081035e0 0x7fb7081a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7081061a0 0x7fb7081030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb6f40a3d40 0x7fb6f40a6200 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 --2- 172.21.15.32:0/4250881573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb708101120 0x7fb708102b60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 >> 172.21.15.32:0/4250881573 conn(0x7fb708077fa0 msgr2=0x7fb708100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 shutdown_connections 2024-04-23T18:20:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:40.954+0000 7fb70ff76640 1 -- 172.21.15.32:0/4250881573 wait complete. 2024-04-23T18:20:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:40 smithi040 ceph-mon[32626]: pgmap v170: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 55 MiB/s wr, 2.33k op/s 2024-04-23T18:20:41.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:40 smithi064 ceph-mon[33062]: pgmap v170: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 55 MiB/s wr, 2.33k op/s 2024-04-23T18:20:41.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:40 smithi032 ceph-mon[29221]: pgmap v170: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 55 MiB/s wr, 2.33k op/s 2024-04-23T18:20:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:42 smithi040 ceph-mon[32626]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 39 MiB/s wr, 1.99k op/s 2024-04-23T18:20:43.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:42 smithi064 ceph-mon[33062]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 39 MiB/s wr, 1.99k op/s 2024-04-23T18:20:43.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:42 smithi032 ceph-mon[29221]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 39 MiB/s wr, 1.99k op/s 2024-04-23T18:20:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 315.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:20:45.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:44 smithi040 ceph-mon[32626]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 39 MiB/s wr, 2.00k op/s 2024-04-23T18:20:45.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:44 smithi064 ceph-mon[33062]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 39 MiB/s wr, 2.00k op/s 2024-04-23T18:20:45.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:44 smithi032 ceph-mon[29221]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 39 MiB/s wr, 2.00k op/s 2024-04-23T18:20:47.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:46 smithi040 ceph-mon[32626]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 935 op/s 2024-04-23T18:20:47.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:46 smithi064 ceph-mon[33062]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 935 op/s 2024-04-23T18:20:47.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:46 smithi032 ceph-mon[29221]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 935 op/s 2024-04-23T18:20:48.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 320.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:20:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:48 smithi040 ceph-mon[32626]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 6.5 MiB/s wr, 533 op/s 2024-04-23T18:20:49.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:48 smithi064 ceph-mon[33062]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 6.5 MiB/s wr, 533 op/s 2024-04-23T18:20:49.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:48 smithi032 ceph-mon[29221]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 6.5 MiB/s wr, 533 op/s 2024-04-23T18:20:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:50 smithi040 ceph-mon[32626]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-23T18:20:51.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:50 smithi064 ceph-mon[33062]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-23T18:20:51.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:50 smithi032 ceph-mon[29221]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-23T18:20:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:52 smithi040 ceph-mon[32626]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-23T18:20:53.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:52 smithi064 ceph-mon[33062]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-23T18:20:53.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:52 smithi032 ceph-mon[29221]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-23T18:20:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 325.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:20:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 -- 172.21.15.32:0/4233930271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe0101120 msgr2=0x7febe0101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 --2- 172.21.15.32:0/4233930271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe0101120 0x7febe0101500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7febdc009900 tx=0x7febdc02a410 comp rx=0 tx=0).stop 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 -- 172.21.15.32:0/4233930271 shutdown_connections 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 --2- 172.21.15.32:0/4233930271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe01061a0 0x7febe010dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 --2- 172.21.15.32:0/4233930271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe0101ad0 0x7febe0105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 --2- 172.21.15.32:0/4233930271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe0101120 0x7febe0101500 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 -- 172.21.15.32:0/4233930271 >> 172.21.15.32:0/4233930271 conn(0x7febe0077fa0 msgr2=0x7febe00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 -- 172.21.15.32:0/4233930271 shutdown_connections 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 -- 172.21.15.32:0/4233930271 wait complete. 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.053+0000 7febe7d8e640 1 Processor -- start 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 -- start start 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0101120 0x7febe01995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe019a080 0x7febe01a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febe0111110 con 0x7febe01061a0 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7febe0110f90 con 0x7febe019a080 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe7d8e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febe0111290 con 0x7febe0101120 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47766/0 (socket says 172.21.15.32:47766) 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 -- 172.21.15.32:0/2654675300 learned_addr learned my addr 172.21.15.32:0/2654675300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe632b640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe019a080 0x7febe01a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5b2a640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0101120 0x7febe01995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 -- 172.21.15.32:0/2654675300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0101120 msgr2=0x7febe01995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0101120 0x7febe01995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 -- 172.21.15.32:0/2654675300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe019a080 msgr2=0x7febe01a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe019a080 0x7febe01a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 -- 172.21.15.32:0/2654675300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe01a4e00 con 0x7febe01061a0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.054+0000 7febe5329640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7febd000e9c0 tx=0x7febd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.055+0000 7febd6ffd640 1 -- 172.21.15.32:0/2654675300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd000cd80 con 0x7febe01061a0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.055+0000 7febd6ffd640 1 -- 172.21.15.32:0/2654675300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febd00054d0 con 0x7febe01061a0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.055+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe01a50f0 con 0x7febe01061a0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.055+0000 7febd6ffd640 1 -- 172.21.15.32:0/2654675300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0010430 con 0x7febe01061a0 2024-04-23T18:20:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.055+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe01a5640 con 0x7febe01061a0 2024-04-23T18:20:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.058+0000 7febd6ffd640 1 -- 172.21.15.32:0/2654675300 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7febd0004fd0 con 0x7febe01061a0 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.058+0000 7febeab7d740 1 Processor -- start 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.058+0000 7febeab7d740 1 -- start start 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d8a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72d9c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f72cef970 con 0x559f72d8a500 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f72c8c4a0 con 0x559f72d99c10 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febeab7d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559f72ceb930 con 0x559f72d8ae20 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d8a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febd6ffd640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 0x7febcc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5329640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72d9c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5329640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72d9c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:43588/0 (socket says 172.21.15.32:43588) 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49850/0 (socket says 172.21.15.32:49850) 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5329640 1 -- 172.21.15.32:0/1588174147 learned_addr learned my addr 172.21.15.32:0/1588174147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe632b640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 0x7febcc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febd6ffd640 1 -- 172.21.15.32:0/2654675300 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7febd00d0c60 con 0x7febe01061a0 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 -- 172.21.15.32:0/1588174147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 msgr2=0x559f72d9c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72d9c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 -- 172.21.15.32:0/1588174147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 msgr2=0x559f72d8a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d8a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe5b2a640 1 -- 172.21.15.32:0/1588174147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f72a02210 con 0x559f72d8ae20 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.059+0000 7febe632b640 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d8a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.060+0000 7febe632b640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 0x7febcc0a61d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7febd8009fd0 tx=0x7febd8004ab0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.060+0000 7febe5b2a640 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7febdc009230 tx=0x7febdc032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965883bfc103bb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.060+0000 7febcbfff640 1 -- 172.21.15.32:0/1588174147 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc03f070 con 0x559f72d8ae20 2024-04-23T18:20:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.060+0000 7febcbfff640 1 -- 172.21.15.32:0/1588174147 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febdc032e40 con 0x559f72d8ae20 2024-04-23T18:20:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.060+0000 7febcbfff640 1 -- 172.21.15.32:0/1588174147 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc035070 con 0x559f72d8ae20 2024-04-23T18:20:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- 172.21.15.32:0/1588174147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 msgr2=0x559f72d996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7febdc009230 tx=0x7febdc032b90 comp rx=0 tx=0).stop 2024-04-23T18:20:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- 172.21.15.32:0/1588174147 shutdown_connections 2024-04-23T18:20:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72d9c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8ae20 0x559f72d996d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 --2- 172.21.15.32:0/1588174147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d8a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- 172.21.15.32:0/1588174147 >> 172.21.15.32:0/1588174147 conn(0x559f72d86b30 msgr2=0x559f72d86f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- 172.21.15.32:0/1588174147 shutdown_connections 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- 172.21.15.32:0/1588174147 wait complete. 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 -- start start 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.061+0000 7febeab7d740 1 Processor -- start 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d96860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d99c10 0x559f72d96dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f72d9c7e0 con 0x559f72d8a500 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f72c8a0b0 con 0x559f72d8ae20 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559f72d9c960 con 0x559f72d99c10 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febeab7d740 10 client.0 _ll_register_callbacks cb 0x559f72c77d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47770/0 (socket says 172.21.15.32:47770) 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 -- 172.21.15.32:0/478648067 learned_addr learned my addr 172.21.15.32:0/478648067 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5329640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d96860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe632b640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d99c10 0x559f72d96dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 -- 172.21.15.32:0/478648067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d99c10 msgr2=0x559f72d96dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d99c10 0x559f72d96dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 -- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 msgr2=0x559f72d96860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d96860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.062+0000 7febe5b2a640 1 -- 172.21.15.32:0/478648067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f72ace620 con 0x559f72d8a500 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febe5b2a640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7febdc031440 tx=0x7febdc031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edbd5ab89e44008 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc03f070 con 0x559f72d8a500 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7febdc004440 con 0x559f72d8a500 2024-04-23T18:20:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559f72cfd390 con 0x559f72d8a500 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7febdc054050 con 0x559f72d8a500 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 10 client.16197 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f72da8790 con 0x559f72d8a500 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559f72cfd390 con 0x559f72d8a500 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7febdc004880 con 0x559f72d8a500 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 10 client.16197 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 20 client.16197 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72da9ae0 0x559f72dabed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 4 client.16197 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.063+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f72cf7200 con 0x559f72da9ae0 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.064+0000 7febe5329640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72da9ae0 0x559f72dabed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.064+0000 7febe5329640 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72da9ae0 0x559f72dabed0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f5272097e3369b server_cookie=9637812cd327c3a3 in_seq=0 out_seq=0 2024-04-23T18:20:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.064+0000 7febc97fa640 10 client.16197 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febc97fa640 1 -- 172.21.15.32:0/478648067 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f72cf7200 con 0x559f72da9ae0 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febc97fa640 10 client.16197 handle_command_reply: tid=0 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 client.16197 shutdown 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 msgr2=0x559f72d99570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7febdc031440 tx=0x7febdc031750 comp rx=0 tx=0).stop 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 shutdown_connections 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72da9ae0 0x559f72dabed0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d99c10 0x559f72d96dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d96860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febc97fa640 0 client.16197 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 --2- 172.21.15.32:0/478648067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8a500 0x559f72d99570 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 >> 172.21.15.32:0/478648067 conn(0x559f72d86b30 msgr2=0x559f72d8c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 shutdown_connections 2024-04-23T18:20:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 1 -- 172.21.15.32:0/478648067 wait complete. 2024-04-23T18:20:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.065+0000 7febeab7d740 20 client.16197 trim_cache size 0 max 16384 2024-04-23T18:20:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.080+0000 7febeab7d740 1 Processor -- start 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.080+0000 7febeab7d740 1 -- start start 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d85b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72db1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febe01ae180 con 0x559f72d8ae20 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7febe01ab9c0 con 0x559f72d99c10 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febeab7d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febe01a9810 con 0x559f72d8a500 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47774/0 (socket says 172.21.15.32:47774) 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 -- 172.21.15.32:0/274790999 learned_addr learned my addr 172.21.15.32:0/274790999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5b2a640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d85b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 -- 172.21.15.32:0/274790999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 msgr2=0x559f72d85b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d85b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 -- 172.21.15.32:0/274790999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 msgr2=0x559f72db1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe632b640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72db1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72db1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 -- 172.21.15.32:0/274790999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f72d87700 con 0x559f72d8ae20 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe632b640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72db1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febe5329640 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7febd00990b0 tx=0x7febd00d3170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c8297aed1ecf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febcb7fe640 1 -- 172.21.15.32:0/274790999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd00046c0 con 0x559f72d8ae20 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febcb7fe640 1 -- 172.21.15.32:0/274790999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febd009d350 con 0x559f72d8ae20 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.081+0000 7febcb7fe640 1 -- 172.21.15.32:0/274790999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd009af50 con 0x559f72d8ae20 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- 172.21.15.32:0/274790999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 msgr2=0x559f72d990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7febd00990b0 tx=0x7febd00d3170 comp rx=0 tx=0).stop 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- 172.21.15.32:0/274790999 shutdown_connections 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d99c10 0x559f72db1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d8ae20 0x559f72d990a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 --2- 172.21.15.32:0/274790999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d85b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- 172.21.15.32:0/274790999 >> 172.21.15.32:0/274790999 conn(0x559f72d86b30 msgr2=0x559f72d85420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- 172.21.15.32:0/274790999 shutdown_connections 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- 172.21.15.32:0/274790999 wait complete. 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 -- start start 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.082+0000 7febeab7d740 1 Processor -- start 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d9d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d9db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febe01a9e80 con 0x559f72d99c10 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7febe01ab2f0 con 0x559f72d8ae20 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febe01aa550 con 0x559f72d8a500 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febeab7d740 10 client.0 _ll_register_callbacks cb 0x7febd80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47786/0 (socket says 172.21.15.32:47786) 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 -- 172.21.15.32:0/147905988 learned_addr learned my addr 172.21.15.32:0/147905988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe5329640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d9db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe5b2a640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d9d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 -- 172.21.15.32:0/147905988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 msgr2=0x559f72d9d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d9d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 -- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 msgr2=0x559f72d9db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d9db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.083+0000 7febe632b640 1 -- 172.21.15.32:0/147905988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f72e0edd0 con 0x559f72d99c10 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febe632b640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7febd8013a60 tx=0x7febd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdab7cce347b991 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8004b20 con 0x559f72d99c10 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7febd801c070 con 0x559f72d99c10 2024-04-23T18:20:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febd009d350 con 0x559f72d99c10 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7febd8004cc0 con 0x559f72d99c10 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 10 client.16203 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f72e10020 con 0x559f72d99c10 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febd009d350 con 0x559f72d99c10 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7febd8029410 con 0x559f72d99c10 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 10 client.16203 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 20 client.16203 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72e10850 0x559f72e12c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 4 client.16203 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.084+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f72c735f0 con 0x559f72e10850 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.085+0000 7febe5b2a640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72e10850 0x559f72e12c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.085+0000 7febe5b2a640 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72e10850 0x559f72e12c70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb9bc2336f7f58d server_cookie=5b05c732c49c11b5 in_seq=0 out_seq=0 2024-04-23T18:20:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.085+0000 7febc97fa640 10 client.16203 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.096+0000 7febc97fa640 1 -- 172.21.15.32:0/147905988 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f72c735f0 con 0x559f72e10850 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.096+0000 7febc97fa640 10 client.16203 handle_command_reply: tid=0 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.096+0000 7febeab7d740 1 client.16203 shutdown 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.096+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 msgr2=0x559f72d9e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.096+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7febd8013a60 tx=0x7febd8004a40 comp rx=0 tx=0).stop 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 shutdown_connections 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f72e10850 0x559f72e12c70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f72d99c10 0x559f72d9e070 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f72d8ae20 0x559f72d9db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febc97fa640 0 client.16203 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 --2- 172.21.15.32:0/147905988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f72d8a500 0x559f72d9d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 >> 172.21.15.32:0/147905988 conn(0x559f72d86b30 msgr2=0x559f72d85420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 shutdown_connections 2024-04-23T18:20:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 1 -- 172.21.15.32:0/147905988 wait complete. 2024-04-23T18:20:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febeab7d740 20 client.16203 trim_cache size 0 max 16384 2024-04-23T18:20:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:20:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:20:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:20:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 msgr2=0x7febcc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 0x7febcc0a61d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7febd8009fd0 tx=0x7febd8004ab0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 msgr2=0x7febe0199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7febd000e9c0 tx=0x7febd000ee80 comp rx=0 tx=0).stop 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 shutdown_connections 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febe019a080 0x7febe01a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7febcc0a3d10 0x7febcc0a61d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febe01061a0 0x7febe0199af0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 --2- 172.21.15.32:0/2654675300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0101120 0x7febe01995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:54.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.097+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 >> 172.21.15.32:0/2654675300 conn(0x7febe0077fa0 msgr2=0x7febe00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:54.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.098+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 shutdown_connections 2024-04-23T18:20:54.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:20:54.098+0000 7febe7d8e640 1 -- 172.21.15.32:0/2654675300 wait complete. 2024-04-23T18:20:55.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:54 smithi040 ceph-mon[32626]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-23T18:20:55.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:54 smithi064 ceph-mon[33062]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-23T18:20:55.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:54 smithi032 ceph-mon[29221]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-23T18:20:57.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:56 smithi040 ceph-mon[32626]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:20:57.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:56 smithi064 ceph-mon[33062]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:20:57.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:56 smithi032 ceph-mon[29221]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:20:57.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4003405812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4106ed0 msgr2=0x7f9aa41072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4003405812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41072b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009900 tx=0x7f9aa002a410 comp rx=0 tx=0).stop 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4003405812 shutdown_connections 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4003405812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 0x7f9aa410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4003405812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa41010a0 0x7f9aa4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4003405812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41072b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4003405812 >> 172.21.15.32:0/4003405812 conn(0x7f9aa4077c40 msgr2=0x7f9aa4078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4003405812 shutdown_connections 2024-04-23T18:20:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4003405812 wait complete. 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 Processor -- start 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.906+0000 7f9aac5a2640 1 -- start start 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 0x7f9aa4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41110d0 con 0x7f9aa41010a0 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4110f50 con 0x7f9aa4106ed0 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aac5a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4111250 con 0x7f9aa4101a40 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45002/0 (socket says 172.21.15.32:45002) 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 -- 172.21.15.32:0/4171525588 learned_addr learned my addr 172.21.15.32:0/4171525588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aa9b3d640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 0x7f9aa4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaab3f640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 -- 172.21.15.32:0/4171525588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 msgr2=0x7f9aa4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 0x7f9aa4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 -- 172.21.15.32:0/4171525588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4106ed0 msgr2=0x7f9aa41a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 -- 172.21.15.32:0/4171525588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa41a4b30 con 0x7f9aa41010a0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.907+0000 7f9aaa33e640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0005ab0 tx=0x7f9aa0003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.908+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa003f070 con 0x7f9aa41010a0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.908+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa41a4d00 con 0x7f9aa41010a0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.908+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa0031d60 con 0x7f9aa41010a0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.908+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0035030 con 0x7f9aa41010a0 2024-04-23T18:20:57.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.908+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41a51c0 con 0x7f9aa41010a0 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.909+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa4068e00 con 0x7f9aa41010a0 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.910+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9aa0042070 con 0x7f9aa41010a0 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.910+0000 7f9a9b7fe640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 0x7f9a900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.911+0000 7f9aa9b3d640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 0x7f9a900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.911+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9aa00f10b0 con 0x7f9aa41010a0 2024-04-23T18:20:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.911+0000 7f9aa9b3d640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 0x7f9a900a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4069e00 tx=0x7f9a9400a4c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:57.915+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9aa00bd840 con 0x7f9aa41010a0 2024-04-23T18:20:58.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.141+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9aa4109330 con 0x7f9aa41010a0 2024-04-23T18:20:58.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.141+0000 7f9a9b7fe640 1 -- 172.21.15.32:0/4171525588 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11855 (secure 0 0 0) 0x7f9aa00bd140 con 0x7f9aa41010a0 2024-04-23T18:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:20:38.801047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:20:58.135 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 msgr2=0x7f9a900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 0x7f9a900a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4069e00 tx=0x7f9a9400a4c0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 msgr2=0x7f9aa4199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0005ab0 tx=0x7f9aa0003cd0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 shutdown_connections 2024-04-23T18:20:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4106ed0 0x7f9aa41a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aa4101a40 0x7f9aa4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9a900a3d20 0x7f9a900a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 --2- 172.21.15.32:0/4171525588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa41010a0 0x7f9aa4199570 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.143+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 >> 172.21.15.32:0/4171525588 conn(0x7f9aa4077c40 msgr2=0x7f9aa40ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.144+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 shutdown_connections 2024-04-23T18:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.144+0000 7f9aac5a2640 1 -- 172.21.15.32:0/4171525588 wait complete. 2024-04-23T18:20:58.148 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 9de88f28 --release --await' 2024-04-23T18:20:58.148 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 9de88f28 --release --await 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 -- 172.21.15.32:0/2522310688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c10c6d0 msgr2=0x7fcc8c10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 --2- 172.21.15.32:0/2522310688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c10eb60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009ca0 tx=0x7fcc8402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 -- 172.21.15.32:0/2522310688 shutdown_connections 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 --2- 172.21.15.32:0/2522310688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c10eb60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 --2- 172.21.15.32:0/2522310688 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c103b70 0x7fcc8c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 --2- 172.21.15.32:0/2522310688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 -- 172.21.15.32:0/2522310688 >> 172.21.15.32:0/2522310688 conn(0x7fcc8c077f70 msgr2=0x7fcc8c078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 -- 172.21.15.32:0/2522310688 shutdown_connections 2024-04-23T18:20:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.263+0000 7fcc91f76640 1 -- 172.21.15.32:0/2522310688 wait complete. 2024-04-23T18:20:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 Processor -- start 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 -- start start 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc8c198720 0x7fcc8c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1123a0 con 0x7fcc8c1031c0 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c112220 con 0x7fcc8c198720 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc91f76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c112520 con 0x7fcc8c10c6d0 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45018/0 (socket says 172.21.15.32:45018) 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.264+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/1129925820 learned_addr learned my addr 172.21.15.32:0/1129925820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8affd640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/1129925820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 msgr2=0x7fcc8c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/1129925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc8c198720 msgr2=0x7fcc8c198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc8c198720 0x7fcc8c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/1129925820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8c199320 con 0x7fcc8c1031c0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8affd640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c890 tx=0x7fcc7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc88ff9640 1 -- 172.21.15.32:0/1129925820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c004220 con 0x7fcc8c1031c0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc88ff9640 1 -- 172.21.15.32:0/1129925820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc7c0043c0 con 0x7fcc8c1031c0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc8c1a5400 con 0x7fcc8c1031c0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc88ff9640 1 -- 172.21.15.32:0/1129925820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c004a00 con 0x7fcc8c1031c0 2024-04-23T18:20:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.265+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8c1a5800 con 0x7fcc8c1031c0 2024-04-23T18:20:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.267+0000 7fcc88ff9640 1 -- 172.21.15.32:0/1129925820 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcc7c0079b0 con 0x7fcc8c1031c0 2024-04-23T18:20:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 Processor -- start 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 -- start start 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac15870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a1ab7a6d0 con 0x561a1ac20480 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a1ab171a0 con 0x561a1ac15490 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a1ab76690 con 0x561a1ac15db0 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc88ff9640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 0x7fcc740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45032/0 (socket says 172.21.15.32:45032) 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 -- 172.21.15.32:0/2004091061 learned_addr learned my addr 172.21.15.32:0/2004091061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac15870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 0x7fcc740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc88ff9640 1 -- 172.21.15.32:0/1129925820 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc7c0d0610 con 0x7fcc8c1031c0 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 -- 172.21.15.32:0/2004091061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 msgr2=0x561a1ac1ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 -- 172.21.15.32:0/2004091061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 msgr2=0x561a1ac15870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac15870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.268+0000 7fcc8affd640 1 -- 172.21.15.32:0/2004091061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a1ab215b0 con 0x561a1ac20480 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.269+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009bf0 tx=0x7fcc80010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64084ae4a8ba4267 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.269+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 0x7fcc740a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8402ade0 tx=0x7fcc840095c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.269+0000 7fcc71ffb640 1 -- 172.21.15.32:0/2004091061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc80004020 con 0x561a1ac20480 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.269+0000 7fcc71ffb640 1 -- 172.21.15.32:0/2004091061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc800041c0 con 0x561a1ac20480 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.269+0000 7fcc71ffb640 1 -- 172.21.15.32:0/2004091061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc800126b0 con 0x561a1ac20480 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2004091061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 msgr2=0x561a1ac22870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009bf0 tx=0x7fcc80010740 comp rx=0 tx=0).stop 2024-04-23T18:20:58.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2004091061 shutdown_connections 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac22870 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2004091061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac15870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2004091061 >> 172.21.15.32:0/2004091061 conn(0x561a1ac117b0 msgr2=0x561a1ac11b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2004091061 shutdown_connections 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2004091061 wait complete. 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- start start 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 Processor -- start 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac1fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a1ac23310 con 0x561a1ac20480 2024-04-23T18:20:58.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a1ab14b20 con 0x561a1ac15490 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.270+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a1ac23490 con 0x561a1ac15db0 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc94d6b740 10 client.0 _ll_register_callbacks cb 0x561a1ab14530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45048/0 (socket says 172.21.15.32:45048) 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 -- 172.21.15.32:0/3362645291 learned_addr learned my addr 172.21.15.32:0/3362645291 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac1fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8affd640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 -- 172.21.15.32:0/3362645291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 msgr2=0x561a1ac1d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 -- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 msgr2=0x561a1ac1fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac1fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 -- 172.21.15.32:0/3362645291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a1a88c920 con 0x561a1ac20480 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcc84035000 tx=0x7fcc84005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd520c1935ec61da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc6affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc84050030 con 0x561a1ac20480 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc84031af0 con 0x561a1ac20480 2024-04-23T18:20:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc800041c0 con 0x561a1ac20480 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcc8405a050 con 0x561a1ac20480 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc94d6b740 10 client.16221 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.271+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561a1a958d30 con 0x561a1ac20480 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc800041c0 con 0x561a1ac20480 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fcc84030860 con 0x561a1ac20480 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 10 client.16221 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 10 client.16221 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 20 client.16221 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac30640 0x561a1ac32ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 4 client.16221 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a1ab38500 con 0x561a1ac30640 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac30640 0x561a1ac32ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac30640 0x561a1ac32ac0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aad1835699304 server_cookie=95676a7d9b656592 in_seq=0 out_seq=0 2024-04-23T18:20:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.272+0000 7fcc6affd640 10 client.16221 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc6affd640 1 -- 172.21.15.32:0/3362645291 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a1ab38500 con 0x561a1ac30640 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc6affd640 10 client.16221 handle_command_reply: tid=0 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 client.16221 shutdown 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 msgr2=0x561a1ac1d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcc84035000 tx=0x7fcc84005da0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 shutdown_connections 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac30640 0x561a1ac32ac0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac20480 0x561a1ac1d780 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac1d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc6affd640 0 client.16221 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3362645291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15490 0x561a1ac1fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.273+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 >> 172.21.15.32:0/3362645291 conn(0x561a1ac117b0 msgr2=0x561a1ac175b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.274+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 shutdown_connections 2024-04-23T18:20:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.274+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3362645291 wait complete. 2024-04-23T18:20:58.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.274+0000 7fcc94d6b740 20 client.16221 trim_cache size 0 max 16384 2024-04-23T18:20:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 Processor -- start 2024-04-23T18:20:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 -- start start 2024-04-23T18:20:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 0x561a1ac1f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac20480 0x561a1ac38390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1b0390 con 0x561a1ac15490 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1adbd0 con 0x561a1ac15db0 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1aba40 con 0x561a1ac20480 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45060/0 (socket says 172.21.15.32:45060) 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/3245373884 learned_addr learned my addr 172.21.15.32:0/3245373884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8affd640 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 0x561a1ac1f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/3245373884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac20480 msgr2=0x561a1ac38390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac20480 0x561a1ac38390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/3245373884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 msgr2=0x561a1ac1f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 0x561a1ac1f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/3245373884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a1ac12380 con 0x561a1ac15490 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8affd640 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 0x561a1ac1f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.291+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0cddb0 tx=0x7fcc7c06f090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a76873b2d7688201 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc717fa640 1 -- 172.21.15.32:0/3245373884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c01d040 con 0x561a1ac15490 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc717fa640 1 -- 172.21.15.32:0/3245373884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc7c09cca0 con 0x561a1ac15490 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc717fa640 1 -- 172.21.15.32:0/3245373884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c098c10 con 0x561a1ac15490 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3245373884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 msgr2=0x561a1ac2c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0cddb0 tx=0x7fcc7c06f090 comp rx=0 tx=0).stop 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3245373884 shutdown_connections 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac20480 0x561a1ac38390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac15db0 0x561a1ac1f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/3245373884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac2c1d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3245373884 >> 172.21.15.32:0/3245373884 conn(0x561a1ac117b0 msgr2=0x561a1ac109d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3245373884 shutdown_connections 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- 172.21.15.32:0/3245373884 wait complete. 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.292+0000 7fcc94d6b740 1 -- start start 2024-04-23T18:20:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 Processor -- start 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac28c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 0x561a1ac21790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1ac090 con 0x561a1ac15490 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1ad500 con 0x561a1ac20480 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c1ac760 con 0x561a1ac15db0 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc94d6b740 10 client.0 _ll_register_callbacks cb 0x7fcc84009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45076/0 (socket says 172.21.15.32:45076) 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/2043347545 learned_addr learned my addr 172.21.15.32:0/2043347545 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac28c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 0x561a1ac21790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/2043347545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 msgr2=0x561a1ac28c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac28c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 msgr2=0x561a1ac21790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 0x561a1ac21790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8b7fe640 1 -- 172.21.15.32:0/2043347545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a1ac21cd0 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.293+0000 7fcc8bfff640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 0x561a1ac21790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc8b7fe640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c09c440 tx=0x7fcc7c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a754286a73733 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c01d040 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fcc7c098c10 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc7c09cca0 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcc7c0a5210 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 10 client.16227 fetch_fsmap learned FSMap version 35 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561a1ac97460 con 0x561a1ac15490 2024-04-23T18:20:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc7c09cca0 con 0x561a1ac15490 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fcc7c0bb2a0 con 0x561a1ac15490 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 10 client.16227 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 10 client.16227 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 20 client.16227 populate_metadata read hostname 'smithi032' 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac97c80 0x561a1ac9a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 4 client.16227 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9de88f28", "release": true, "await": true}] 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.294+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9de88f28", "release": true, "await": true}) -- 0x561a1ab880f0 con 0x561a1ac97c80 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.295+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac97c80 0x561a1ac9a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.295+0000 7fcc6affd640 10 client.16227 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.295+0000 7fcc8affd640 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac97c80 0x561a1ac9a0a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb24a26f37b9846b server_cookie=c1f044d902b24926 in_seq=0 out_seq=0 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.303+0000 7fcc6affd640 1 -- 172.21.15.32:0/2043347545 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561a1ab880f0 con 0x561a1ac97c80 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc6affd640 10 client.16227 handle_command_reply: tid=0 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 client.16227 shutdown 2024-04-23T18:20:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 msgr2=0x561a1ac286e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c09c440 tx=0x7fcc7c00c810 comp rx=0 tx=0).stop 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 shutdown_connections 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a1ac97c80 0x561a1ac9a0a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a1ac20480 0x561a1ac21790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a1ac15db0 0x561a1ac28c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc6affd640 0 client.16227 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 --2- 172.21.15.32:0/2043347545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a1ac15490 0x561a1ac286e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 >> 172.21.15.32:0/2043347545 conn(0x561a1ac117b0 msgr2=0x561a1ac2b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 shutdown_connections 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 1 -- 172.21.15.32:0/2043347545 wait complete. 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc94d6b740 20 client.16227 trim_cache size 0 max 16384 2024-04-23T18:20:58.333 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 35, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 20, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "9de88f28": { 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 20, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:20:58.334 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 80.6, 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:20:58.335 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 msgr2=0x7fcc740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 0x7fcc740a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8402ade0 tx=0x7fcc840095c0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 msgr2=0x7fcc8c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c890 tx=0x7fcc7c00cd50 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 shutdown_connections 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc8c198720 0x7fcc8c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc8c10c6d0 0x7fcc8c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fcc740a3d20 0x7fcc740a61e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 --2- 172.21.15.32:0/1129925820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c1031c0 0x7fcc8c19e660 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.304+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 >> 172.21.15.32:0/1129925820 conn(0x7fcc8c077f70 msgr2=0x7fcc8c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.305+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 shutdown_connections 2024-04-23T18:20:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:20:58.305+0000 7fcc91f76640 1 -- 172.21.15.32:0/1129925820 wait complete. 2024-04-23T18:20:58.338 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9de88f28', seconds elapsed: 22.2 2024-04-23T18:20:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 330.0 s, 48.8 tps, lat 8700.936 ms stddev 10696.015 2024-04-23T18:20:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:58 smithi040 ceph-mon[32626]: pgmap v179: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-23T18:20:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:20:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4171525588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:59.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:58 smithi064 ceph-mon[33062]: pgmap v179: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-23T18:20:59.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:20:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4171525588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:59.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:58 smithi032 ceph-mon[29221]: pgmap v179: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-23T18:20:59.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:20:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4171525588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:00 smithi040 ceph-mon[32626]: pgmap v180: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-23T18:21:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:00 smithi064 ceph-mon[33062]: pgmap v180: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-23T18:21:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:00 smithi032 ceph-mon[29221]: pgmap v180: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-23T18:21:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:02 smithi040 ceph-mon[32626]: pgmap v181: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 13 op/s 2024-04-23T18:21:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:02 smithi064 ceph-mon[33062]: pgmap v181: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 13 op/s 2024-04-23T18:21:03.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:02 smithi032 ceph-mon[29221]: pgmap v181: 97 pgs: 97 active+clean; 3.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 13 op/s 2024-04-23T18:21:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 335.0 s, 3495.0 tps, lat 32.247 ms stddev 289.666 2024-04-23T18:21:05.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:04 smithi040 ceph-mon[32626]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.4 MiB/s wr, 736 op/s 2024-04-23T18:21:05.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:04 smithi064 ceph-mon[33062]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.4 MiB/s wr, 736 op/s 2024-04-23T18:21:05.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:04 smithi032 ceph-mon[29221]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.4 MiB/s wr, 736 op/s 2024-04-23T18:21:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:06 smithi040 ceph-mon[32626]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:06 smithi064 ceph-mon[33062]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 -- 172.21.15.32:0/629656347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb974108f60 msgr2=0x7fb974115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 --2- 172.21.15.32:0/629656347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb974108f60 0x7fb974115420 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009ca0 tx=0x7fb96c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 -- 172.21.15.32:0/629656347 shutdown_connections 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 --2- 172.21.15.32:0/629656347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb974108f60 0x7fb974115420 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 --2- 172.21.15.32:0/629656347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9741085a0 0x7fb974108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 --2- 172.21.15.32:0/629656347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97410e660 0x7fb97410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 -- 172.21.15.32:0/629656347 >> 172.21.15.32:0/629656347 conn(0x7fb974076460 msgr2=0x7fb974078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 -- 172.21.15.32:0/629656347 shutdown_connections 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 -- 172.21.15.32:0/629656347 wait complete. 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.247+0000 7fb979413640 1 Processor -- start 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 -- start start 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 0x7fb974073460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb974108f60 0x7fb9740739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 0x7fb97406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 0x7fb974073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 0x7fb974073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46182/0 (socket says 172.21.15.32:46182) 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb97406bf40 con 0x7fb9741085a0 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb972ffd640 1 -- 172.21.15.32:0/2462379373 learned_addr learned my addr 172.21.15.32:0/2462379373 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb97406bdc0 con 0x7fb974108f60 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb97406c0c0 con 0x7fb97410e660 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.248+0000 7fb9727fc640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb974108f60 0x7fb9740739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 0x7fb97406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 -- 172.21.15.32:0/2462379373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb974108f60 msgr2=0x7fb9740739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb974108f60 0x7fb9740739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 -- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 msgr2=0x7fb974073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 0x7fb974073460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.249+0000 7fb9737fe640 1 -- 172.21.15.32:0/2462379373 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97406dd90 con 0x7fb97410e660 2024-04-23T18:21:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb9737fe640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 0x7fb97406d4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009bf0 tx=0x7fb96c0319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb95bfff640 1 -- 172.21.15.32:0/2462379373 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c003b70 con 0x7fb97410e660 2024-04-23T18:21:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb95bfff640 1 -- 172.21.15.32:0/2462379373 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb96c003d10 con 0x7fb97410e660 2024-04-23T18:21:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb95bfff640 1 -- 172.21.15.32:0/2462379373 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c02f970 con 0x7fb97410e660 2024-04-23T18:21:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97406e020 con 0x7fb97410e660 2024-04-23T18:21:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.250+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9741b3fd0 con 0x7fb97410e660 2024-04-23T18:21:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 Processor -- start 2024-04-23T18:21:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb95bfff640 1 -- 172.21.15.32:0/2462379373 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb96c02fb10 con 0x7fb97410e660 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 -- start start 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2f40 0x56055bcad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bcafbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56055bc07970 con 0x56055bcad800 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56055bba44a0 con 0x56055bca2620 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.254+0000 7fb97c209740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56055bc03930 con 0x56055bca2f40 2024-04-23T18:21:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45916/0 (socket says 172.21.15.32:45916) 2024-04-23T18:21:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 -- 172.21.15.32:0/2982219493 learned_addr learned my addr 172.21.15.32:0/2982219493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bcafbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 -- 172.21.15.32:0/2982219493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2f40 msgr2=0x56055bcad2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2f40 0x56055bcad2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 -- 172.21.15.32:0/2982219493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 msgr2=0x56055bcafbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bcafbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb9727fc640 1 -- 172.21.15.32:0/2982219493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055b91a210 con 0x56055bca2620 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.255+0000 7fb95bfff640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 0x7fb95c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb95bfff640 1 -- 172.21.15.32:0/2462379373 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb96c0f19e0 con 0x7fb97410e660 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb972ffd640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 0x7fb95c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.250 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb9727fc640 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb96800bfd0 tx=0x7fb9680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebb9b176a980b3af server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb9597fa640 1 -- 172.21.15.32:0/2982219493 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968012ce0 con 0x56055bca2620 2024-04-23T18:21:07.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb9597fa640 1 -- 172.21.15.32:0/2982219493 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968005020 con 0x56055bca2620 2024-04-23T18:21:07.251 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.256+0000 7fb9597fa640 1 -- 172.21.15.32:0/2982219493 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968004a00 con 0x56055bca2620 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 -- 172.21.15.32:0/2982219493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 msgr2=0x56055bca2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb96800bfd0 tx=0x7fb9680046c0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb972ffd640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 0x7fb95c0a63e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x56055bca3bc0 tx=0x7fb964005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 -- 172.21.15.32:0/2982219493 shutdown_connections 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bcafbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2f40 0x56055bcad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 --2- 172.21.15.32:0/2982219493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2620 0x56055bca2a00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 -- 172.21.15.32:0/2982219493 >> 172.21.15.32:0/2982219493 conn(0x56055bc9eb30 msgr2=0x56055bc9ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.257+0000 7fb97c209740 1 -- 172.21.15.32:0/2982219493 shutdown_connections 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 -- 172.21.15.32:0/2982219493 wait complete. 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 -- start start 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 Processor -- start 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 0x56055bcaa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bcad800 0x56055bcaaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56055bcc1440 con 0x56055bca2f40 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56055bba20b0 con 0x56055bcad800 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.258+0000 7fb97c209740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56055bcc15c0 con 0x56055bca2620 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.259+0000 7fb97c209740 10 client.0 _ll_register_callbacks cb 0x56055bb8fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.259+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.259+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:33676/0 (socket says 172.21.15.32:33676) 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.259+0000 7fb972ffd640 1 -- 172.21.15.32:0/394253529 learned_addr learned my addr 172.21.15.32:0/394253529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:07.254 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.259+0000 7fb9727fc640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 0x56055bcaa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.260+0000 7fb9727fc640 1 -- 172.21.15.32:0/394253529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 msgr2=0x56055bcad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.260+0000 7fb9727fc640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.260+0000 7fb9727fc640 1 -- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bcad800 msgr2=0x56055bcaaca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:21:07.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.260+0000 7fb9727fc640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bcad800 0x56055bcaaca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.255 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.260+0000 7fb9727fc640 1 -- 172.21.15.32:0/394253529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055b9e6620 con 0x56055bca2f40 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.261+0000 7fb9727fc640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 0x56055bcaa560 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb96800bfd0 tx=0x7fb968005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756e04e33e4f94f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.261+0000 7fb956ff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.261+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968012d50 con 0x56055bca2f40 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.261+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb968005020 con 0x56055bca2f40 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.261+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb968004450 con 0x56055bca2f40 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.262+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb968004730 con 0x56055bca2f40 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.262+0000 7fb97c209740 10 client.16239 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.262+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56055bccd490 con 0x56055bca2f40 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968005020 con 0x56055bca2f40 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fb968010bc0 con 0x56055bca2f40 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 10 client.16239 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 10 client.16239 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 20 client.16239 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bccecc0 0x56055bcd1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 4 client.16239 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.263+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055bc15390 con 0x56055bccecc0 2024-04-23T18:21:07.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:06 smithi032 ceph-mon[29221]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.264+0000 7fb972ffd640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bccecc0 0x56055bcd1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.265+0000 7fb956ff5640 10 client.16239 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.265+0000 7fb972ffd640 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bccecc0 0x56055bcd1140 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85279e134359be9e server_cookie=602c0738d46daab9 in_seq=0 out_seq=0 2024-04-23T18:21:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.266+0000 7fb956ff5640 1 -- 172.21.15.32:0/394253529 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56055bc15390 con 0x56055bccecc0 2024-04-23T18:21:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.266+0000 7fb956ff5640 10 client.16239 handle_command_reply: tid=0 2024-04-23T18:21:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.266+0000 7fb97c209740 1 client.16239 shutdown 2024-04-23T18:21:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.266+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 msgr2=0x56055bcaa560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.266+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 0x56055bcaa560 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb96800bfd0 tx=0x7fb968005480 comp rx=0 tx=0).stop 2024-04-23T18:21:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 shutdown_connections 2024-04-23T18:21:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bccecc0 0x56055bcd1140 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bcad800 0x56055bcaaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2f40 0x56055bcaa560 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 --2- 172.21.15.32:0/394253529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb956ff5640 0 client.16239 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 >> 172.21.15.32:0/394253529 conn(0x56055bc9eb30 msgr2=0x56055bca4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 shutdown_connections 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 1 -- 172.21.15.32:0/394253529 wait complete. 2024-04-23T18:21:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.267+0000 7fb97c209740 20 client.16239 trim_cache size 0 max 16384 2024-04-23T18:21:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.297+0000 7fb97c209740 1 Processor -- start 2024-04-23T18:21:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.298+0000 7fb97c209740 1 -- start start 2024-04-23T18:21:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bc9e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bca96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9741b6af0 con 0x56055bcad800 2024-04-23T18:21:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9741b4330 con 0x56055bca2f40 2024-04-23T18:21:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.299+0000 7fb97c209740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb974080fa0 con 0x56055bca2620 2024-04-23T18:21:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:33684/0 (socket says 172.21.15.32:33684) 2024-04-23T18:21:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb972ffd640 1 -- 172.21.15.32:0/537989009 learned_addr learned my addr 172.21.15.32:0/537989009 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bc9e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9737fe640 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bca96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 -- 172.21.15.32:0/537989009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 msgr2=0x56055bcca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 -- 172.21.15.32:0/537989009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 msgr2=0x56055bca96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bca96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 -- 172.21.15.32:0/537989009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055bc99090 con 0x56055bca2f40 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb9727fc640 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bc9e4a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x56055bcad7d0 tx=0x7fb96800bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d3a0a9f0637c4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.301+0000 7fb958ff9640 1 -- 172.21.15.32:0/537989009 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968033030 con 0x56055bca2f40 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb958ff9640 1 -- 172.21.15.32:0/537989009 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968016040 con 0x56055bca2f40 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb958ff9640 1 -- 172.21.15.32:0/537989009 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968003e60 con 0x56055bca2f40 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 -- 172.21.15.32:0/537989009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 msgr2=0x56055bc9e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bc9e4a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x56055bcad7d0 tx=0x7fb96800bf30 comp rx=0 tx=0).stop 2024-04-23T18:21:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 -- 172.21.15.32:0/537989009 shutdown_connections 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bcad800 0x56055bca96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bc9e4a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 --2- 172.21.15.32:0/537989009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bca2620 0x56055bcca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 -- 172.21.15.32:0/537989009 >> 172.21.15.32:0/537989009 conn(0x56055bc9eb30 msgr2=0x56055bc9d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 -- 172.21.15.32:0/537989009 shutdown_connections 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.302+0000 7fb97c209740 1 -- 172.21.15.32:0/537989009 wait complete. 2024-04-23T18:21:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.303+0000 7fb97c209740 1 -- start start 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.303+0000 7fb97c209740 1 Processor -- start 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bcc1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bcad800 0x56055bcc2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9740815f0 con 0x56055bca2620 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb974082a60 con 0x56055bca2f40 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb974081cc0 con 0x56055bcad800 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb97c209740 10 client.0 _ll_register_callbacks cb 0x7fb968005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46212/0 (socket says 172.21.15.32:46212) 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb972ffd640 1 -- 172.21.15.32:0/3025744563 learned_addr learned my addr 172.21.15.32:0/3025744563 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.304+0000 7fb9727fc640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bcc1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 -- 172.21.15.32:0/3025744563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bcad800 msgr2=0x56055bcc2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bcad800 0x56055bcc2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 -- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 msgr2=0x56055bcc1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bcc1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 -- 172.21.15.32:0/3025744563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055bd33270 con 0x56055bca2620 2024-04-23T18:21:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb972ffd640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x56055bcd19b0 tx=0x7fb964004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c25af74abc5ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb956ff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9640291b0 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.305+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb964029350 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.306+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb968016040 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.306+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb964029550 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.306+0000 7fb97c209740 10 client.16245 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.306+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56055bd34540 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.306+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968016040 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7fb964012450 con 0x56055bca2620 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 10 client.16245 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 10 client.16245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 20 client.16245 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bd34d50 0x56055bd37170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 4 client.16245 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.307+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56055bc0f200 con 0x56055bd34d50 2024-04-23T18:21:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.308+0000 7fb9727fc640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bd34d50 0x56055bd37170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.308+0000 7fb956ff5640 10 client.16245 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.309+0000 7fb9727fc640 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bd34d50 0x56055bd37170 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c887b2e7698678 server_cookie=2e05cb3f44aeafb7 in_seq=0 out_seq=0 2024-04-23T18:21:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.373+0000 7fb956ff5640 1 -- 172.21.15.32:0/3025744563 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56055bc0f200 con 0x56055bd34d50 2024-04-23T18:21:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.373+0000 7fb956ff5640 10 client.16245 handle_command_reply: tid=0 2024-04-23T18:21:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.373+0000 7fb97c209740 1 client.16245 shutdown 2024-04-23T18:21:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.374+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 msgr2=0x56055bcc17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.374+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x56055bcd19b0 tx=0x7fb964004730 comp rx=0 tx=0).stop 2024-04-23T18:21:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 shutdown_connections 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56055bd34d50 0x56055bd37170 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56055bcad800 0x56055bcc2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56055bca2f40 0x56055bcc1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 --2- 172.21.15.32:0/3025744563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56055bca2620 0x56055bcc17d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 >> 172.21.15.32:0/3025744563 conn(0x56055bc9eb30 msgr2=0x56055bc9dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb956ff5640 0 client.16245 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 shutdown_connections 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 1 -- 172.21.15.32:0/3025744563 wait complete. 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.375+0000 7fb97c209740 20 client.16245 trim_cache size 0 max 16384 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:21:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:21:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:21:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.376+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 msgr2=0x7fb95c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.376+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 0x7fb95c0a63e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x56055bca3bc0 tx=0x7fb964005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 msgr2=0x7fb97406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 0x7fb97406d4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009bf0 tx=0x7fb96c0319f0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 shutdown_connections 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb97410e660 0x7fb97406d4d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb974108f60 0x7fb9740739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb95c0a3f20 0x7fb95c0a63e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 --2- 172.21.15.32:0/2462379373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9741085a0 0x7fb974073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 >> 172.21.15.32:0/2462379373 conn(0x7fb974076460 msgr2=0x7fb974077d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 shutdown_connections 2024-04-23T18:21:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:07.377+0000 7fb979413640 1 -- 172.21.15.32:0/2462379373 wait complete. 2024-04-23T18:21:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 340.0 s, 3109.4 tps, lat 32.024 ms stddev 36.942 2024-04-23T18:21:09.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:08 smithi040 ceph-mon[32626]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:09.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:08 smithi064 ceph-mon[33062]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:09.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:08 smithi032 ceph-mon[29221]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-23T18:21:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:10 smithi064 ceph-mon[33062]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-04-23T18:21:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:10 smithi032 ceph-mon[29221]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-04-23T18:21:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:10 smithi040 ceph-mon[32626]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-04-23T18:21:13.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:12 smithi064 ceph-mon[33062]: pgmap v186: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-23T18:21:13.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:12 smithi032 ceph-mon[29221]: pgmap v186: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-23T18:21:13.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:12 smithi040 ceph-mon[32626]: pgmap v186: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-23T18:21:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 345.0 s, 2717.0 tps, lat 37.034 ms stddev 39.798 2024-04-23T18:21:15.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:14 smithi064 ceph-mon[33062]: pgmap v187: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.61k op/s 2024-04-23T18:21:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:14 smithi032 ceph-mon[29221]: pgmap v187: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.61k op/s 2024-04-23T18:21:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:14 smithi040 ceph-mon[32626]: pgmap v187: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.61k op/s 2024-04-23T18:21:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:16 smithi064 ceph-mon[33062]: pgmap v188: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-23T18:21:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:16 smithi032 ceph-mon[29221]: pgmap v188: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-23T18:21:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:16 smithi040 ceph-mon[32626]: pgmap v188: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-23T18:21:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 350.0 s, 3393.8 tps, lat 29.415 ms stddev 32.669 2024-04-23T18:21:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:18 smithi064 ceph-mon[33062]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 762 op/s 2024-04-23T18:21:19.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:18 smithi032 ceph-mon[29221]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 762 op/s 2024-04-23T18:21:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:18 smithi040 ceph-mon[32626]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 762 op/s 2024-04-23T18:21:20.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:19 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:20.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:19 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:20.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:19 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:20.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.521+0000 7f4786b8e640 1 -- 172.21.15.32:0/3949455351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47801085a0 msgr2=0x7f4780108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.521+0000 7f4786b8e640 1 --2- 172.21.15.32:0/3949455351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47801085a0 0x7f4780108a20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4774009900 tx=0x7f477402a400 comp rx=0 tx=0).stop 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.522+0000 7f4786b8e640 1 -- 172.21.15.32:0/3949455351 shutdown_connections 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.522+0000 7f4786b8e640 1 --2- 172.21.15.32:0/3949455351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780108f60 0x7f4780115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.522+0000 7f4786b8e640 1 --2- 172.21.15.32:0/3949455351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47801085a0 0x7f4780108a20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.522+0000 7f4786b8e640 1 --2- 172.21.15.32:0/3949455351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f478010e660 0x7f478010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.522+0000 7f4786b8e640 1 -- 172.21.15.32:0/3949455351 >> 172.21.15.32:0/3949455351 conn(0x7f4780076460 msgr2=0x7f4780078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.523+0000 7f4786b8e640 1 -- 172.21.15.32:0/3949455351 shutdown_connections 2024-04-23T18:21:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.523+0000 7f4786b8e640 1 -- 172.21.15.32:0/3949455351 wait complete. 2024-04-23T18:21:20.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.524+0000 7f4786b8e640 1 Processor -- start 2024-04-23T18:21:20.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.524+0000 7f4786b8e640 1 -- start start 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f4786b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 0x7f478019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f4786b8e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 0x7f478019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f4786b8e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 0x7f47801a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f478492a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 0x7f478019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f478492a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 0x7f478019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:60788/0 (socket says 172.21.15.32:60788) 2024-04-23T18:21:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 0x7f478019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 0x7f478019f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55166/0 (socket says 172.21.15.32:55166) 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.525+0000 7f478492a640 1 -- 172.21.15.32:0/1547126845 learned_addr learned my addr 172.21.15.32:0/1547126845 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47801188a0 con 0x7f4780108f60 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4780118720 con 0x7f47801085a0 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4780118a20 con 0x7f478010e660 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 0x7f47801a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 -- 172.21.15.32:0/1547126845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 msgr2=0x7f478019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 0x7f478019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 -- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 msgr2=0x7f478019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 0x7f478019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.526+0000 7f478512b640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47801a3d70 con 0x7f478010e660 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.527+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 0x7f47801a3640 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f477800be20 tx=0x7f477800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.528+0000 7f477dffb640 1 -- 172.21.15.32:0/1547126845 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778019070 con 0x7f478010e660 2024-04-23T18:21:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.528+0000 7f477dffb640 1 -- 172.21.15.32:0/1547126845 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47780148f0 con 0x7f478010e660 2024-04-23T18:21:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.528+0000 7f477dffb640 1 -- 172.21.15.32:0/1547126845 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778012840 con 0x7f478010e660 2024-04-23T18:21:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.528+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47801a4060 con 0x7f478010e660 2024-04-23T18:21:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.528+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47801ab8f0 con 0x7f478010e660 2024-04-23T18:21:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.532+0000 7f477dffb640 1 -- 172.21.15.32:0/1547126845 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f477801c070 con 0x7f478010e660 2024-04-23T18:21:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.532+0000 7f478997d740 1 Processor -- start 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.532+0000 7f478997d740 1 -- start start 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199620 0x5602ed199a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 0x5602ed1a6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5602ed0fe970 con 0x5602ed1a4800 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5602ed09b4a0 con 0x5602ed199f40 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478997d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ed0fa930 con 0x5602ed199620 2024-04-23T18:21:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477dffb640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 0x7f47680a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 0x5602ed1a6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 0x5602ed1a6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55172/0 (socket says 172.21.15.32:55172) 2024-04-23T18:21:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477ffff640 1 -- 172.21.15.32:0/1612359830 learned_addr learned my addr 172.21.15.32:0/1612359830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477ffff640 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199620 0x5602ed199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f477dffb640 1 -- 172.21.15.32:0/1547126845 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f47780cfda0 con 0x7f478010e660 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 0x7f47680a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 -- 172.21.15.32:0/1612359830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199620 msgr2=0x5602ed199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199620 0x5602ed199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 -- 172.21.15.32:0/1612359830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 msgr2=0x5602ed1a6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 0x5602ed1a6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.533+0000 7f478492a640 1 -- 172.21.15.32:0/1612359830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ece11210 con 0x5602ed199f40 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.534+0000 7f478492a640 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a42c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4770009c80 tx=0x7f4770011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f27420ab824cfee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.534+0000 7f478512b640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 0x7f47680a63a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47780a4580 tx=0x7f477800e040 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.534+0000 7f4766ffd640 1 -- 172.21.15.32:0/1612359830 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770004230 con 0x5602ed199f40 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.534+0000 7f4766ffd640 1 -- 172.21.15.32:0/1612359830 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47700043d0 con 0x5602ed199f40 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.534+0000 7f4766ffd640 1 -- 172.21.15.32:0/1612359830 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770004d40 con 0x5602ed199f40 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 -- 172.21.15.32:0/1612359830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 msgr2=0x5602ed1a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a42c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4770009c80 tx=0x7f4770011810 comp rx=0 tx=0).stop 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 -- 172.21.15.32:0/1612359830 shutdown_connections 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed1a4800 0x5602ed1a6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a42c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 --2- 172.21.15.32:0/1612359830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199620 0x5602ed199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 -- 172.21.15.32:0/1612359830 >> 172.21.15.32:0/1612359830 conn(0x5602ed195b30 msgr2=0x5602ed195f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 -- 172.21.15.32:0/1612359830 shutdown_connections 2024-04-23T18:21:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.535+0000 7f478997d740 1 -- 172.21.15.32:0/1612359830 wait complete. 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 -- start start 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 Processor -- start 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5602ed1a7430 con 0x5602ed199620 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5602ed0990b0 con 0x5602ed199f40 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ed1a75b0 con 0x5602ed1a4800 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478997d740 10 client.0 _ll_register_callbacks cb 0x5602ed086d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55186/0 (socket says 172.21.15.32:55186) 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f477ffff640 1 -- 172.21.15.32:0/2462844586 learned_addr learned my addr 172.21.15.32:0/2462844586 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f478492a640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.536+0000 7f477ffff640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 -- 172.21.15.32:0/2462844586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 msgr2=0x5602ed1a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 -- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 msgr2=0x5602ed1a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 -- 172.21.15.32:0/2462844586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ecedd620 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f478492a640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f477ffff640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4774004650 tx=0x7f477402a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8cd30764b33e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f475ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774035070 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47700043d0 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4774003800 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4774003a00 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f478997d740 10 client.16257 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.537+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5602ed1b33e0 con 0x5602ed199620 2024-04-23T18:21:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47700043d0 con 0x5602ed199620 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f477403d450 con 0x5602ed199620 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 10 client.16257 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 10 client.16257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 20 client.16257 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed1b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 4 client.16257 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602ed10c390 con 0x5602ed1b4890 2024-04-23T18:21:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.538+0000 7f478492a640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed1b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.540+0000 7f478492a640 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed1b6cc0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3cca74212c2dae1 server_cookie=54feac1be3b171e0 in_seq=0 out_seq=0 2024-04-23T18:21:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.540+0000 7f475ffff640 10 client.16257 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f475ffff640 1 -- 172.21.15.32:0/2462844586 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602ed10c390 con 0x5602ed1b4890 2024-04-23T18:21:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f475ffff640 10 client.16257 handle_command_reply: tid=0 2024-04-23T18:21:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 client.16257 shutdown 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 msgr2=0x5602ed1a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4774004650 tx=0x7f477402a400 comp rx=0 tx=0).stop 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 shutdown_connections 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed1b6cc0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f475ffff640 0 client.16257 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199f40 0x5602ed1a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 --2- 172.21.15.32:0/2462844586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1a41a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.541+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 >> 172.21.15.32:0/2462844586 conn(0x5602ed195b30 msgr2=0x5602ed19b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.542+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 shutdown_connections 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.542+0000 7f478997d740 1 -- 172.21.15.32:0/2462844586 wait complete. 2024-04-23T18:21:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.542+0000 7f478997d740 20 client.16257 trim_cache size 0 max 16384 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 Processor -- start 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 -- start start 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 0x5602ed194b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed1a4800 0x5602ed1bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b6940 con 0x5602ed199620 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b4180 con 0x5602ed1a4800 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.564+0000 7f478997d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b1fd0 con 0x5602ed199f40 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55192/0 (socket says 172.21.15.32:55192) 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 -- 172.21.15.32:0/3027119964 learned_addr learned my addr 172.21.15.32:0/3027119964 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f478492a640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 0x5602ed194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed1a4800 0x5602ed1bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 -- 172.21.15.32:0/3027119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 msgr2=0x5602ed194b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 0x5602ed194b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 -- 172.21.15.32:0/3027119964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed1a4800 msgr2=0x5602ed1bc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed1a4800 0x5602ed1bc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f477ffff640 1 -- 172.21.15.32:0/3027119964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ed1a0440 con 0x5602ed199620 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.565+0000 7f478492a640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 0x5602ed194b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.566+0000 7f477ffff640 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x5602ed1b0870 tx=0x7f4774005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd406094a615bea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.566+0000 7f47667fc640 1 -- 172.21.15.32:0/3027119964 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774035070 con 0x5602ed199620 2024-04-23T18:21:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.566+0000 7f47667fc640 1 -- 172.21.15.32:0/3027119964 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477403d650 con 0x5602ed199620 2024-04-23T18:21:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.566+0000 7f47667fc640 1 -- 172.21.15.32:0/3027119964 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774004920 con 0x5602ed199620 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 -- 172.21.15.32:0/3027119964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 msgr2=0x5602ed1b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x5602ed1b0870 tx=0x7f4774005e80 comp rx=0 tx=0).stop 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 -- 172.21.15.32:0/3027119964 shutdown_connections 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed1a4800 0x5602ed1bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed199f40 0x5602ed194b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 --2- 172.21.15.32:0/3027119964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199620 0x5602ed1b02f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 -- 172.21.15.32:0/3027119964 >> 172.21.15.32:0/3027119964 conn(0x5602ed195b30 msgr2=0x5602ed194420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 -- 172.21.15.32:0/3027119964 shutdown_connections 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.567+0000 7f478997d740 1 -- 172.21.15.32:0/3027119964 wait complete. 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478997d740 1 -- start start 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478997d740 1 Processor -- start 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478997d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478997d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478492a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f477ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f477ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:60818/0 (socket says 172.21.15.32:60818) 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.568+0000 7f478492a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55202/0 (socket says 172.21.15.32:55202) 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f478997d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1aa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 -- 172.21.15.32:0/2502213940 learned_addr learned my addr 172.21.15.32:0/2502213940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b2640 con 0x5602ed199f40 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b3ab0 con 0x5602ed199620 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f47801b2d10 con 0x5602ed1a4800 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 -- 172.21.15.32:0/2502213940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 msgr2=0x5602ed1aa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1aa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 -- 172.21.15.32:0/2502213940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 msgr2=0x5602ed1a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f477ffff640 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ed199d70 con 0x5602ed199620 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.569+0000 7f478492a640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.570+0000 7f477ffff640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4774005400 tx=0x7f4774005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22223eb863feef7f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.570+0000 7f475ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.570+0000 7f478997d740 10 client.0 _ll_register_callbacks cb 0x7f47740052f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f477403d650 con 0x5602ed199620 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f475ffff640 1 -- 172.21.15.32:0/2502213940 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774035070 con 0x5602ed199620 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f475ffff640 1 -- 172.21.15.32:0/2502213940 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f477403d450 con 0x5602ed199620 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f475ffff640 1 -- 172.21.15.32:0/2502213940 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4774004140 con 0x5602ed199620 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f478997d740 10 client.25865 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.571+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5602ed21a920 con 0x5602ed199620 2024-04-23T18:21:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f475ffff640 1 -- 172.21.15.32:0/2502213940 <== mon.2 v2:172.21.15.40:3300/0 4 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f47740048f0 con 0x5602ed199620 2024-04-23T18:21:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 10 client.25865 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 20 client.25865 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed21cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 4 client.25865 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f477403d650 con 0x5602ed1b4890 2024-04-23T18:21:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478492a640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed21cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f475ffff640 10 client.25865 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.572+0000 7f478492a640 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed21cd70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b26597e00e184aee server_cookie=312f0563c75a70a8 in_seq=0 out_seq=0 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f475ffff640 1 -- 172.21.15.32:0/2502213940 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f477403d650 con 0x5602ed1b4890 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f475ffff640 10 client.25865 handle_command_reply: tid=0 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 client.25865 shutdown 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 msgr2=0x5602ed1acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4774005400 tx=0x7f4774005dd0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 shutdown_connections 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5602ed1b4890 0x5602ed21cd70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ed1a4800 0x5602ed1aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602ed199f40 0x5602ed1a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 --2- 172.21.15.32:0/2502213940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602ed199620 0x5602ed1acab0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 >> 172.21.15.32:0/2502213940 conn(0x5602ed195b30 msgr2=0x5602ed194cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f475ffff640 0 client.25865 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 shutdown_connections 2024-04-23T18:21:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 1 -- 172.21.15.32:0/2502213940 wait complete. 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.585+0000 7f478997d740 20 client.25865 trim_cache size 0 max 16384 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.587+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 msgr2=0x7f47680a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.587+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 0x7f47680a63a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47780a4580 tx=0x7f477800e040 comp rx=0 tx=0).stop 2024-04-23T18:21:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.587+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 msgr2=0x7f47801a3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.587+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 0x7f47801a3640 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f477800be20 tx=0x7f477800bf10 comp rx=0 tx=0).stop 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 shutdown_connections 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f478010e660 0x7f47801a3640 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f47680a3ee0 0x7f47680a63a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4780108f60 0x7f478019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 --2- 172.21.15.32:0/1547126845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47801085a0 0x7f478019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 >> 172.21.15.32:0/1547126845 conn(0x7f4780076460 msgr2=0x7f4780077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 shutdown_connections 2024-04-23T18:21:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:20.588+0000 7f4786b8e640 1 -- 172.21.15.32:0/1547126845 wait complete. 2024-04-23T18:21:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:20 smithi064 ceph-mon[33062]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-23T18:21:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:20 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:20 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:20 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:21:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:20 smithi032 ceph-mon[29221]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-23T18:21:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:20 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:20 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:20 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:21:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:20 smithi040 ceph-mon[32626]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-23T18:21:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:20 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:20 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:20 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:21:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:22 smithi064 ceph-mon[33062]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-04-23T18:21:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:22 smithi032 ceph-mon[29221]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-04-23T18:21:23.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:22 smithi040 ceph-mon[32626]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-04-23T18:21:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 355.0 s, 3506.0 tps, lat 28.486 ms stddev 31.953 2024-04-23T18:21:25.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:24 smithi064 ceph-mon[33062]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.6 MiB/s wr, 772 op/s 2024-04-23T18:21:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:24 smithi032 ceph-mon[29221]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.6 MiB/s wr, 772 op/s 2024-04-23T18:21:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:24 smithi040 ceph-mon[32626]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.6 MiB/s wr, 772 op/s 2024-04-23T18:21:27.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:26 smithi064 ceph-mon[33062]: pgmap v193: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-23T18:21:27.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:26 smithi032 ceph-mon[29221]: pgmap v193: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-23T18:21:27.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:26 smithi040 ceph-mon[32626]: pgmap v193: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-23T18:21:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 360.0 s, 2806.8 tps, lat 35.697 ms stddev 37.810 2024-04-23T18:21:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:28 smithi064 ceph-mon[33062]: pgmap v194: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.8 MiB/s wr, 630 op/s 2024-04-23T18:21:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:28 smithi032 ceph-mon[29221]: pgmap v194: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.8 MiB/s wr, 630 op/s 2024-04-23T18:21:29.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:28 smithi040 ceph-mon[32626]: pgmap v194: 97 pgs: 97 active+clean; 3.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.8 MiB/s wr, 630 op/s 2024-04-23T18:21:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:30 smithi064 ceph-mon[33062]: pgmap v195: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 914 op/s 2024-04-23T18:21:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:30 smithi032 ceph-mon[29221]: pgmap v195: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 914 op/s 2024-04-23T18:21:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:30 smithi040 ceph-mon[32626]: pgmap v195: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 914 op/s 2024-04-23T18:21:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:32 smithi064 ceph-mon[33062]: pgmap v196: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-23T18:21:33.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:32 smithi032 ceph-mon[29221]: pgmap v196: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-23T18:21:33.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:32 smithi040 ceph-mon[32626]: pgmap v196: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 -- 172.21.15.32:0/936970888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f791010e6c0 msgr2=0x7f791010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 --2- 172.21.15.32:0/936970888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f791010e6c0 0x7f791010eaa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f790c009930 tx=0x7f790c02a420 comp rx=0 tx=0).stop 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 -- 172.21.15.32:0/936970888 shutdown_connections 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 --2- 172.21.15.32:0/936970888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7910109750 0x7f7910115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 --2- 172.21.15.32:0/936970888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910108d90 0x7f7910109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 --2- 172.21.15.32:0/936970888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f791010e6c0 0x7f791010eaa0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.723+0000 7f7917613640 1 -- 172.21.15.32:0/936970888 >> 172.21.15.32:0/936970888 conn(0x7f7910077ec0 msgr2=0x7f791007a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.724+0000 7f7917613640 1 -- 172.21.15.32:0/936970888 shutdown_connections 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.724+0000 7f7917613640 1 -- 172.21.15.32:0/936970888 wait complete. 2024-04-23T18:21:33.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.724+0000 7f7917613640 1 Processor -- start 2024-04-23T18:21:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.725+0000 7f7917613640 1 -- start start 2024-04-23T18:21:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 0x7f791019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f791010e6c0 0x7f79101a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79101187f0 con 0x7f7910108d90 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7910118670 con 0x7f7910109750 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7917613640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7910118970 con 0x7f791010e6c0 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 0x7f791019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:39068/0 (socket says 172.21.15.32:39068) 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 -- 172.21.15.32:0/865323927 learned_addr learned my addr 172.21.15.32:0/865323927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7915bb0640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f791010e6c0 0x7f79101a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 -- 172.21.15.32:0/865323927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f791010e6c0 msgr2=0x7f79101a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f791010e6c0 0x7f79101a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 -- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 msgr2=0x7f791019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 0x7f791019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 -- 172.21.15.32:0/865323927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79101a3de0 con 0x7f7910109750 2024-04-23T18:21:33.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f79153af640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 0x7f791019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.726+0000 7f7914bae640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f790000e9f0 tx=0x7f790000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.727+0000 7f79067fc640 1 -- 172.21.15.32:0/865323927 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790000cd80 con 0x7f7910109750 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.727+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79101a40d0 con 0x7f7910109750 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.727+0000 7f79067fc640 1 -- 172.21.15.32:0/865323927 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79000108e0 con 0x7f7910109750 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.727+0000 7f79067fc640 1 -- 172.21.15.32:0/865323927 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900018980 con 0x7f7910109750 2024-04-23T18:21:33.722 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.727+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79101ab960 con 0x7f7910109750 2024-04-23T18:21:33.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.730+0000 7f79067fc640 1 -- 172.21.15.32:0/865323927 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7900010430 con 0x7f7910109750 2024-04-23T18:21:33.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.730+0000 7f791a407740 1 Processor -- start 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 -- start start 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 0x557eb85aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557eb8511970 con 0x557eb85acf40 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557eb84ae4a0 con 0x557eb85ac620 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.731+0000 7f791a407740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557eb850d930 con 0x557eb85b7800 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7915bb0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 0x557eb85aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40380/0 (socket says 172.21.15.32:40380) 2024-04-23T18:21:33.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40140/0 (socket says 172.21.15.32:40140) 2024-04-23T18:21:33.726 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 -- 172.21.15.32:0/640581013 learned_addr learned my addr 172.21.15.32:0/640581013 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 -- 172.21.15.32:0/640581013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 msgr2=0x557eb85b9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 -- 172.21.15.32:0/640581013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 msgr2=0x557eb85aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 0x557eb85aca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 -- 172.21.15.32:0/640581013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eb8224210 con 0x557eb85acf40 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7914bae640 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7915bb0640 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 0x557eb85aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79067fc640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 0x7f78fc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79067fc640 1 -- 172.21.15.32:0/865323927 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f79000d0ba0 con 0x7f7910109750 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f79153af640 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f790c006010 tx=0x7f790c032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd0a18354f652fb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.732+0000 7f7915bb0640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 0x7f78fc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f78fb7fe640 1 -- 172.21.15.32:0/640581013 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c035040 con 0x557eb85acf40 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f78fb7fe640 1 -- 172.21.15.32:0/640581013 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c031860 con 0x557eb85acf40 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f78fb7fe640 1 -- 172.21.15.32:0/640581013 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c004a00 con 0x557eb85acf40 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f7915bb0640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 0x7f78fc0a61d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7908009fd0 tx=0x7f7908009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 -- 172.21.15.32:0/640581013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 msgr2=0x557eb85b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f790c006010 tx=0x7f790c032ed0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 -- 172.21.15.32:0/640581013 shutdown_connections 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85b7800 0x557eb85b9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85acf40 0x557eb85b72c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.733+0000 7f791a407740 1 --2- 172.21.15.32:0/640581013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85ac620 0x557eb85aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.734+0000 7f791a407740 1 -- 172.21.15.32:0/640581013 >> 172.21.15.32:0/640581013 conn(0x557eb85a8b30 msgr2=0x557eb85a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.734+0000 7f791a407740 1 -- 172.21.15.32:0/640581013 shutdown_connections 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.734+0000 7f791a407740 1 -- 172.21.15.32:0/640581013 wait complete. 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.734+0000 7f791a407740 1 -- start start 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.734+0000 7f791a407740 1 Processor -- start 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557eb85ba3d0 con 0x557eb85ac620 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557eb84ac0b0 con 0x557eb85b7800 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557eb85ba550 con 0x557eb85acf40 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f791a407740 10 client.0 _ll_register_callbacks cb 0x557eb8499d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40142/0 (socket says 172.21.15.32:40142) 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 -- 172.21.15.32:0/1926328442 learned_addr learned my addr 172.21.15.32:0/1926328442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7915bb0640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 -- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 msgr2=0x557eb85b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 -- 172.21.15.32:0/1926328442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 msgr2=0x557eb85b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7914bae640 1 -- 172.21.15.32:0/1926328442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eb82f0620 con 0x557eb85acf40 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.735+0000 7f7915bb0640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f7914bae640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f790000cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=405c750deaa10704 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f78f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:21:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900010720 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f79000991a0 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790c031860 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f790009de60 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f791a407740 10 client.25861 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.737+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557eb85c62c0 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c031860 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f79000ce560 con 0x557eb85acf40 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 10 client.25861 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 10 client.25861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 20 client.25861 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb85c7780 0x557eb85c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 4 client.25861 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557eb851f390 con 0x557eb85c7780 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.738+0000 7f79153af640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb85c7780 0x557eb85c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.739+0000 7f79153af640 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb85c7780 0x557eb85c9b90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=924871f25b426931 server_cookie=68d84649fa7de32e in_seq=0 out_seq=0 2024-04-23T18:21:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.739+0000 7f78f8ff9640 10 client.25861 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.739+0000 7f78f8ff9640 1 -- 172.21.15.32:0/1926328442 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557eb851f390 con 0x557eb85c7780 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.739+0000 7f78f8ff9640 10 client.25861 handle_command_reply: tid=0 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 client.25861 shutdown 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 msgr2=0x557eb85b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f790000cf20 comp rx=0 tx=0).stop 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 shutdown_connections 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb85c7780 0x557eb85c9b90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85b43c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 --2- 172.21.15.32:0/1926328442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 >> 172.21.15.32:0/1926328442 conn(0x557eb85a8b30 msgr2=0x557eb85ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f78f8ff9640 0 client.25861 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 shutdown_connections 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 1 -- 172.21.15.32:0/1926328442 wait complete. 2024-04-23T18:21:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.740+0000 7f791a407740 20 client.25861 trim_cache size 0 max 16384 2024-04-23T18:21:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.760+0000 7f791a407740 1 Processor -- start 2024-04-23T18:21:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.760+0000 7f791a407740 1 -- start start 2024-04-23T18:21:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.761+0000 7f791a407740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.761+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.761+0000 7f79153af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40394/0 (socket says 172.21.15.32:40394) 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.761+0000 7f791a407740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.761+0000 7f79153af640 1 -- 172.21.15.32:0/2568872824 learned_addr learned my addr 172.21.15.32:0/2568872824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f791a407740 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f7914bae640 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b6b00 con 0x557eb85ac620 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b4340 con 0x557eb85b7800 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b2190 con 0x557eb85acf40 2024-04-23T18:21:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f7915bb0640 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 -- 172.21.15.32:0/2568872824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 msgr2=0x557eb85cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 -- 172.21.15.32:0/2568872824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 msgr2=0x557eb85cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 -- 172.21.15.32:0/2568872824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eb85a8430 con 0x557eb85ac620 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f79153af640 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f790c004650 tx=0x7f790c0058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d70328a0c4bebcc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f78faffd640 1 -- 172.21.15.32:0/2568872824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c035040 con 0x557eb85ac620 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f78faffd640 1 -- 172.21.15.32:0/2568872824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c031760 con 0x557eb85ac620 2024-04-23T18:21:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.762+0000 7f78faffd640 1 -- 172.21.15.32:0/2568872824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c051810 con 0x557eb85ac620 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 msgr2=0x557eb85b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f790c004650 tx=0x7f790c0058f0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 shutdown_connections 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 --2- 172.21.15.32:0/2568872824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85b6c00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 >> 172.21.15.32:0/2568872824 conn(0x557eb85a8b30 msgr2=0x557eb85b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 shutdown_connections 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.763+0000 7f791a407740 1 -- 172.21.15.32:0/2568872824 wait complete. 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.764+0000 7f791a407740 1 -- start start 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.764+0000 7f791a407740 1 Processor -- start 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85bf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85bcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b2800 con 0x557eb85ac620 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b3c70 con 0x557eb85b7800 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f79101b2ed0 con 0x557eb85acf40 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f791a407740 10 client.0 _ll_register_callbacks cb 0x7f7900009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f7915bb0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85bcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f7915bb0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:39114/0 (socket says 172.21.15.32:39114) 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f7914bae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85bcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40168/0 (socket says 172.21.15.32:40168) 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.765+0000 7f7915bb0640 1 -- 172.21.15.32:0/2792870481 learned_addr learned my addr 172.21.15.32:0/2792870481 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f79153af640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85bf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 -- 172.21.15.32:0/2792870481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 msgr2=0x557eb85bcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85bcbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 -- 172.21.15.32:0/2792870481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 msgr2=0x557eb85bf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85bf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 -- 172.21.15.32:0/2792870481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eb85bd630 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f7915bb0640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7908011b70 tx=0x7f7908004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2b99a3903957a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f78f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790801fc70 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f790802a400 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790c031760 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7908038050 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f791a407740 10 client.25883 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.766+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557eb862da10 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c031760 con 0x557eb85b7800 2024-04-23T18:21:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f790801d520 con 0x557eb85b7800 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 10 client.25883 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 10 client.25883 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 20 client.25883 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb862e240 0x557eb8630660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 4 client.25883 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557eb8519200 con 0x557eb862e240 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.767+0000 7f79153af640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb862e240 0x557eb8630660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.768+0000 7f79153af640 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb862e240 0x557eb8630660 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e48b92357c405a9 server_cookie=6e772457e60b42fe in_seq=0 out_seq=0 2024-04-23T18:21:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.768+0000 7f78f8ff9640 10 client.25883 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f78f8ff9640 1 -- 172.21.15.32:0/2792870481 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557eb8519200 con 0x557eb862e240 2024-04-23T18:21:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f78f8ff9640 10 client.25883 handle_command_reply: tid=0 2024-04-23T18:21:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f791a407740 1 client.25883 shutdown 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 msgr2=0x557eb85bd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7908011b70 tx=0x7f7908004ab0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.782+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 shutdown_connections 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557eb862e240 0x557eb8630660 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557eb85b7800 0x557eb85bd0f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557eb85acf40 0x557eb85bcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 --2- 172.21.15.32:0/2792870481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557eb85ac620 0x557eb85bf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f78f8ff9640 0 client.25883 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 >> 172.21.15.32:0/2792870481 conn(0x557eb85a8b30 msgr2=0x557eb85a7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 shutdown_connections 2024-04-23T18:21:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 1 -- 172.21.15.32:0/2792870481 wait complete. 2024-04-23T18:21:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.783+0000 7f791a407740 20 client.25883 trim_cache size 0 max 16384 2024-04-23T18:21:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:21:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:21:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:21:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:21:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.785+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 msgr2=0x7f78fc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.785+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 0x7f78fc0a61d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7908009fd0 tx=0x7f7908009500 comp rx=0 tx=0).stop 2024-04-23T18:21:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.785+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 msgr2=0x7f791019f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.785+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f790000e9f0 tx=0x7f790000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 shutdown_connections 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f791010e6c0 0x7f79101a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7910109750 0x7f791019f330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f78fc0a3d10 0x7f78fc0a61d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 --2- 172.21.15.32:0/865323927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7910108d90 0x7f791019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 >> 172.21.15.32:0/865323927 conn(0x7f7910077ec0 msgr2=0x7f791007a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 shutdown_connections 2024-04-23T18:21:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:33.786+0000 7f7917613640 1 -- 172.21.15.32:0/865323927 wait complete. 2024-04-23T18:21:33.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 365.0 s, 2764.8 tps, lat 36.220 ms stddev 38.809 2024-04-23T18:21:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:34 smithi064 ceph-mon[33062]: pgmap v197: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.2 MiB/s wr, 735 op/s 2024-04-23T18:21:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:34 smithi032 ceph-mon[29221]: pgmap v197: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.2 MiB/s wr, 735 op/s 2024-04-23T18:21:35.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:34 smithi040 ceph-mon[32626]: pgmap v197: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.2 MiB/s wr, 735 op/s 2024-04-23T18:21:36.595 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:21:36.596 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:21:36.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/969905550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca0069820 msgr2=0x7f7ca0069c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/969905550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca0069820 0x7f7ca0069c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c9002a3f0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/969905550 shutdown_connections 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/969905550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca010e7a0 0x7f7ca0110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/969905550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca0069820 0x7f7ca0069c80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/969905550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0102ec0 0x7f7ca0069250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/969905550 >> 172.21.15.32:0/969905550 conn(0x7f7ca0077c40 msgr2=0x7f7ca0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/969905550 shutdown_connections 2024-04-23T18:21:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.746+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/969905550 wait complete. 2024-04-23T18:21:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.747+0000 7f7ca7ffe640 1 Processor -- start 2024-04-23T18:21:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.748+0000 7f7ca7ffe640 1 -- start start 2024-04-23T18:21:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.748+0000 7f7ca7ffe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 0x7f7ca01a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.748+0000 7f7ca7ffe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 0x7f7ca01a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.748+0000 7f7ca5d9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 0x7f7ca01a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5d9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 0x7f7ca01a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:42416/0 (socket says 172.21.15.32:42416) 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca7ffe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca019a6b0 0x7f7ca019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca7ffe640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01143b0 con 0x7f7ca019a6b0 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5d9a640 1 -- 172.21.15.32:0/156484945 learned_addr learned my addr 172.21.15.32:0/156484945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0114230 con 0x7f7ca0069820 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0114530 con 0x7f7ca010e7a0 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 0x7f7ca01a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca659b640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca019a6b0 0x7f7ca019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 -- 172.21.15.32:0/156484945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 msgr2=0x7f7ca01a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 0x7f7ca01a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 -- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca019a6b0 msgr2=0x7f7ca019ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca019a6b0 0x7f7ca019ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.749+0000 7f7ca5599640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca019b2b0 con 0x7f7ca010e7a0 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.750+0000 7f7ca5599640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 0x7f7ca01a0b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009230 tx=0x7f7c900314a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.750+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90035030 con 0x7f7ca010e7a0 2024-04-23T18:21:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.750+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c90003cb0 con 0x7f7ca010e7a0 2024-04-23T18:21:36.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.750+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca019b540 con 0x7f7ca010e7a0 2024-04-23T18:21:36.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.751+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca01a75c0 con 0x7f7ca010e7a0 2024-04-23T18:21:36.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.751+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c900318a0 con 0x7f7ca010e7a0 2024-04-23T18:21:36.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.754+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c84005290 con 0x7f7ca010e7a0 2024-04-23T18:21:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.757+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7c9003a3c0 con 0x7f7ca010e7a0 2024-04-23T18:21:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.758+0000 7f7c96ffd640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 0x7f7c8c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.758+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7c90003840 con 0x7f7ca010e7a0 2024-04-23T18:21:36.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.767+0000 7f7ca5d9a640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 0x7f7c8c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.772+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c90041020 con 0x7f7ca010e7a0 2024-04-23T18:21:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:36.776+0000 7f7ca5d9a640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 0x7f7c8c0a63b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009c90 tx=0x7f7c9c005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.163+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c84005850 con 0x7f7ca010e7a0 2024-04-23T18:21:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.164+0000 7f7c96ffd640 1 -- 172.21.15.32:0/156484945 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11855 (secure 0 0 0) 0x7f7c900bd950 con 0x7f7ca010e7a0 2024-04-23T18:21:37.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:21:37.158 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:20:38.801047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:21:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 msgr2=0x7f7c8c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 0x7f7c8c0a63b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009c90 tx=0x7f7c9c005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 msgr2=0x7f7ca01a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 0x7f7ca01a0b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009230 tx=0x7f7c900314a0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 shutdown_connections 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7c8c0a3ef0 0x7f7c8c0a63b0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009c90 tx=0x7f7c9c005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca019a6b0 0x7f7ca019ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca010e7a0 0x7f7ca01a0b30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 --2- 172.21.15.32:0/156484945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0069820 0x7f7ca01a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.167+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 >> 172.21.15.32:0/156484945 conn(0x7f7ca0077c40 msgr2=0x7f7ca010f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.168+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 shutdown_connections 2024-04-23T18:21:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.168+0000 7f7ca7ffe640 1 -- 172.21.15.32:0/156484945 wait complete. 2024-04-23T18:21:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:36 smithi064 ceph-mon[33062]: pgmap v198: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.0 MiB/s wr, 714 op/s 2024-04-23T18:21:37.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:36 smithi032 ceph-mon[29221]: pgmap v198: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.0 MiB/s wr, 714 op/s 2024-04-23T18:21:37.182 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:21:37.182 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:21:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 -- 172.21.15.32:0/765510338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 msgr2=0x7f8028100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 --2- 172.21.15.32:0/765510338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028100b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8018009cd0 tx=0x7f801802ea70 comp rx=0 tx=0).stop 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 -- 172.21.15.32:0/765510338 shutdown_connections 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 --2- 172.21.15.32:0/765510338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028100b70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 --2- 172.21.15.32:0/765510338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 --2- 172.21.15.32:0/765510338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 0x7f8028106620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 -- 172.21.15.32:0/765510338 >> 172.21.15.32:0/765510338 conn(0x7f80280fc2b0 msgr2=0x7f80280fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 -- 172.21.15.32:0/765510338 shutdown_connections 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.361+0000 7f802d537640 1 -- 172.21.15.32:0/765510338 wait complete. 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 Processor -- start 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 -- start start 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 0x7f802819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8028110070 con 0x7f8028068570 2024-04-23T18:21:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f802810fef0 con 0x7f8028106240 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f802d537640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80281101f0 con 0x7f80281006c0 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f8026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f8026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43684/0 (socket says 172.21.15.32:43684) 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f8026ffd640 1 -- 172.21.15.32:0/1691510353 learned_addr learned my addr 172.21.15.32:0/1691510353 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.362+0000 7f80267fc640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f80277fe640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 0x7f802819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 -- 172.21.15.32:0/1691510353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 msgr2=0x7f8028196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 -- 172.21.15.32:0/1691510353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 msgr2=0x7f802819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 0x7f802819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 -- 172.21.15.32:0/1691510353 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802819b500 con 0x7f8028068570 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f80267fc640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f8026ffd640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8020007860 tx=0x7f8020007c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f801ffff640 1 -- 172.21.15.32:0/1691510353 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020019020 con 0x7f8028068570 2024-04-23T18:21:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f801ffff640 1 -- 172.21.15.32:0/1691510353 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8020015470 con 0x7f8028068570 2024-04-23T18:21:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.363+0000 7f801ffff640 1 -- 172.21.15.32:0/1691510353 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020014630 con 0x7f8028068570 2024-04-23T18:21:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.364+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802819b7f0 con 0x7f8028068570 2024-04-23T18:21:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.365+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80281a3080 con 0x7f8028068570 2024-04-23T18:21:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 Processor -- start 2024-04-23T18:21:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 -- start start 2024-04-23T18:21:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4832370 con 0x55c8a48cd9c0 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c8a47cf2e0 con 0x55c8a48cd0a0 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.368+0000 7f8030250740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c8a482e330 con 0x55c8a48d8090 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53926/0 (socket says 172.21.15.32:53926) 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 -- 172.21.15.32:0/3421978829 learned_addr learned my addr 172.21.15.32:0/3421978829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 -- 172.21.15.32:0/3421978829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 msgr2=0x55c8a48cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 -- 172.21.15.32:0/3421978829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 msgr2=0x55c8a48d7b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:21:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d7b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 -- 172.21.15.32:0/3421978829 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a4610d80 con 0x55c8a48d8090 2024-04-23T18:21:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f801ffff640 1 -- 172.21.15.32:0/1691510353 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f802001c020 con 0x7f8028068570 2024-04-23T18:21:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.369+0000 7f80267fc640 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8010009db0 tx=0x7f801000e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb1cf16fc5ed9c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f801d7fa640 1 -- 172.21.15.32:0/3421978829 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010017070 con 0x55c8a48d8090 2024-04-23T18:21:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f801d7fa640 1 -- 172.21.15.32:0/3421978829 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8010013920 con 0x55c8a48d8090 2024-04-23T18:21:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f801d7fa640 1 -- 172.21.15.32:0/3421978829 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80100127c0 con 0x55c8a48d8090 2024-04-23T18:21:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f801ffff640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 0x7f80140a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f801ffff640 1 -- 172.21.15.32:0/1691510353 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f80200d02f0 con 0x7f8028068570 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f80277fe640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 0x7f80140a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f8030250740 1 -- 172.21.15.32:0/3421978829 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 msgr2=0x55c8a48da480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.370+0000 7f8030250740 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8010009db0 tx=0x7f801000e5a0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 -- 172.21.15.32:0/3421978829 shutdown_connections 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48da480 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- 172.21.15.32:0/3421978829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 -- 172.21.15.32:0/3421978829 >> 172.21.15.32:0/3421978829 conn(0x55c8a48c93c0 msgr2=0x55c8a48c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f80277fe640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 0x7f80140a63b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8018004af0 tx=0x7f801803e040 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 -- 172.21.15.32:0/3421978829 shutdown_connections 2024-04-23T18:21:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 -- 172.21.15.32:0/3421978829 wait complete. 2024-04-23T18:21:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 -- start start 2024-04-23T18:21:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 Processor -- start 2024-04-23T18:21:37.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:36 smithi040 ceph-mon[32626]: pgmap v198: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.0 MiB/s wr, 714 op/s 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48d5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.371+0000 7f8030250740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48d5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f8030250740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c8a48dae80 con 0x55c8a48cd9c0 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f8030250740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c8a47ccfb0 con 0x55c8a48cd0a0 2024-04-23T18:21:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f8030250740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c8a48db000 con 0x55c8a48d8090 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f8030250740 10 client.0 _ll_register_callbacks cb 0x55c8a47cc9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f80277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48d5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f80267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f80267fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43688/0 (socket says 172.21.15.32:43688) 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f80267fc640 1 -- 172.21.15.32:0/686054390 learned_addr learned my addr 172.21.15.32:0/686054390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.372+0000 7f8026ffd640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.373+0000 7f80267fc640 1 -- 172.21.15.32:0/686054390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 msgr2=0x55c8a48d5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.373+0000 7f80267fc640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48d5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.373+0000 7f80267fc640 1 -- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 msgr2=0x55c8a48d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.373+0000 7f80267fc640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48d5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.373+0000 7f80267fc640 1 -- 172.21.15.32:0/686054390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a4671510 con 0x55c8a48cd9c0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.374+0000 7f80267fc640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8010004700 tx=0x7f80100136a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc73316c6b999d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.374+0000 7f800e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.374+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010017040 con 0x55c8a48cd9c0 2024-04-23T18:21:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.374+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8010012950 con 0x55c8a48cd9c0 2024-04-23T18:21:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.374+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8010013920 con 0x55c8a48cd9c0 2024-04-23T18:21:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.375+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8010012b70 con 0x55c8a48cd9c0 2024-04-23T18:21:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.376+0000 7f8030250740 10 client.16293 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.376+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c8a4504670 con 0x55c8a48cd9c0 2024-04-23T18:21:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.376+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8010013920 con 0x55c8a48cd9c0 2024-04-23T18:21:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.376+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f8010023070 con 0x55c8a48cd9c0 2024-04-23T18:21:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.376+0000 7f8030250740 10 client.16293 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8030250740 10 client.16293 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8030250740 20 client.16293 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a48e6710 0x55c8a48e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8030250740 4 client.16293 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8a483fd90 con 0x55c8a48e6710 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.377+0000 7f8026ffd640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a48e6710 0x55c8a48e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.378+0000 7f8026ffd640 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a48e6710 0x55c8a48e8b30 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530693d7dd8fa033 server_cookie=6f93fd78fb7deaac in_seq=0 out_seq=0 2024-04-23T18:21:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.378+0000 7f800e7fc640 10 client.16293 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.379+0000 7f800e7fc640 1 -- 172.21.15.32:0/686054390 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8a483fd90 con 0x55c8a48e6710 2024-04-23T18:21:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.379+0000 7f800e7fc640 10 client.16293 handle_command_reply: tid=0 2024-04-23T18:21:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.379+0000 7f8030250740 1 client.16293 shutdown 2024-04-23T18:21:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 msgr2=0x55c8a48d55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8010004700 tx=0x7f80100136a0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 shutdown_connections 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a48e6710 0x55c8a48e8b30 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f800e7fc640 0 client.16293 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48d5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd9c0 0x55c8a48d55b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 --2- 172.21.15.32:0/686054390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd0a0 0x55c8a48d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.380+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 >> 172.21.15.32:0/686054390 conn(0x55c8a48c93c0 msgr2=0x55c8a48c8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.381+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 shutdown_connections 2024-04-23T18:21:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.381+0000 7f8030250740 1 -- 172.21.15.32:0/686054390 wait complete. 2024-04-23T18:21:37.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.381+0000 7f8030250740 20 client.16293 trim_cache size 0 max 16384 2024-04-23T18:21:37.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.419+0000 7f8030250740 1 Processor -- start 2024-04-23T18:21:37.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.423+0000 7f8030250740 1 -- start start 2024-04-23T18:21:37.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.423+0000 7f8030250740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8030250740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8030250740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8030250740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80281ae330 con 0x55c8a48cd0a0 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43704/0 (socket says 172.21.15.32:43704) 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8026ffd640 1 -- 172.21.15.32:0/3481587200 learned_addr learned my addr 172.21.15.32:0/3481587200 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80281abb70 con 0x55c8a48cd9c0 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80281a99c0 con 0x55c8a48d8090 2024-04-23T18:21:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.424+0000 7f80267fc640 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.425+0000 7f8026ffd640 1 -- 172.21.15.32:0/3481587200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 msgr2=0x55c8a48f4e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:21:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.425+0000 7f8026ffd640 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.425+0000 7f8026ffd640 1 -- 172.21.15.32:0/3481587200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 msgr2=0x55c8a48f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.425+0000 7f8026ffd640 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.425+0000 7f8026ffd640 1 -- 172.21.15.32:0/3481587200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a48db000 con 0x55c8a48cd0a0 2024-04-23T18:21:37.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.431+0000 7f8026ffd640 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f802000f500 tx=0x7f8020015c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188e291f58235c72 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.433+0000 7f801cff9640 1 -- 172.21.15.32:0/3481587200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80200ba070 con 0x55c8a48cd0a0 2024-04-23T18:21:37.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.433+0000 7f801cff9640 1 -- 172.21.15.32:0/3481587200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80200cf040 con 0x55c8a48cd0a0 2024-04-23T18:21:37.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.433+0000 7f801cff9640 1 -- 172.21.15.32:0/3481587200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80200b0070 con 0x55c8a48cd0a0 2024-04-23T18:21:37.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.434+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 msgr2=0x55c8a48c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.434+0000 7f8030250740 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f802000f500 tx=0x7f8020015c80 comp rx=0 tx=0).stop 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.435+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 shutdown_connections 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.435+0000 7f8030250740 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.435+0000 7f8030250740 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.435+0000 7f8030250740 1 --2- 172.21.15.32:0/3481587200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48c8480 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.435+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 >> 172.21.15.32:0/3481587200 conn(0x55c8a48c93c0 msgr2=0x55c8a48c7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.436+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 shutdown_connections 2024-04-23T18:21:37.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.436+0000 7f8030250740 1 -- 172.21.15.32:0/3481587200 wait complete. 2024-04-23T18:21:37.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.437+0000 7f8030250740 1 -- start start 2024-04-23T18:21:37.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.437+0000 7f8030250740 1 Processor -- start 2024-04-23T18:21:37.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.437+0000 7f8030250740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f8030250740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f8030250740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f80267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f80267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:42472/0 (socket says 172.21.15.32:42472) 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f80267fc640 1 -- 172.21.15.32:0/1967993600 learned_addr learned my addr 172.21.15.32:0/1967993600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80281aa030 con 0x55c8a48cd0a0 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f80277fe640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80281ab4a0 con 0x55c8a48cd9c0 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.438+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80281aa700 con 0x55c8a48d8090 2024-04-23T18:21:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f8030250740 10 client.0 _ll_register_callbacks cb 0x7f80100099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 -- 172.21.15.32:0/1967993600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 msgr2=0x55c8a48f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 -- 172.21.15.32:0/1967993600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 msgr2=0x55c8a48f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a48f24d0 con 0x55c8a48cd9c0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f80267fc640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55c8a48d48d0 tx=0x7f8010012f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a308429e5aa1bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.439+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80200cf040 con 0x55c8a48cd9c0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f800e7fc640 10 client.25904 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f800e7fc640 1 -- 172.21.15.32:0/1967993600 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010034060 con 0x55c8a48cd9c0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f800e7fc640 1 -- 172.21.15.32:0/1967993600 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8010024c80 con 0x55c8a48cd9c0 2024-04-23T18:21:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f800e7fc640 1 -- 172.21.15.32:0/1967993600 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8010017050 con 0x55c8a48cd9c0 2024-04-23T18:21:37.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f8030250740 10 client.25904 fetch_fsmap learned FSMap version 35 2024-04-23T18:21:37.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.440+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55c8a494eb60 con 0x55c8a48cd9c0 2024-04-23T18:21:37.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.441+0000 7f800e7fc640 1 -- 172.21.15.32:0/1967993600 <== mon.2 v2:172.21.15.40:3300/0 4 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f8010023040 con 0x55c8a48cd9c0 2024-04-23T18:21:37.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.442+0000 7f8030250740 10 client.25904 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:21:37.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.442+0000 7f8030250740 10 client.25904 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:21:37.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.442+0000 7f8030250740 20 client.25904 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:37.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.442+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a494f170 0x55c8a4951590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.442+0000 7f8030250740 4 client.25904 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:21:37.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.443+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x7f80200cf040 con 0x55c8a494f170 2024-04-23T18:21:37.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.444+0000 7f8026ffd640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a494f170 0x55c8a4951590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.446+0000 7f800e7fc640 10 client.25904 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:37.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.446+0000 7f8026ffd640 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a494f170 0x55c8a4951590 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43cbd1b6e53c4e64 server_cookie=816f767af3db3f38 in_seq=0 out_seq=0 2024-04-23T18:21:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f800e7fc640 1 -- 172.21.15.32:0/1967993600 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f80200cf040 con 0x55c8a494f170 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f800e7fc640 10 client.25904 handle_command_reply: tid=0 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 client.25904 shutdown 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 msgr2=0x55c8a48f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55c8a48d48d0 tx=0x7f8010012f40 comp rx=0 tx=0).stop 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 shutdown_connections 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8a494f170 0x55c8a4951590 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8a48d8090 0x55c8a48f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8a48cd9c0 0x55c8a48f4bf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 --2- 172.21.15.32:0/1967993600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8a48cd0a0 0x55c8a48f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f800e7fc640 0 client.25904 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 >> 172.21.15.32:0/1967993600 conn(0x55c8a48c93c0 msgr2=0x55c8a48e3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 shutdown_connections 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 1 -- 172.21.15.32:0/1967993600 wait complete. 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.667+0000 7f8030250740 20 client.25904 trim_cache size 0 max 16384 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 35, 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 22, 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "a028eb5d": { 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 22, 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:21:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:21:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:21:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:21:37.665 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:21:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 msgr2=0x7f80140a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 0x7f80140a63b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8018004af0 tx=0x7f801803e040 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 msgr2=0x7f8028196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8020007860 tx=0x7f8020007c60 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 shutdown_connections 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8028106240 0x7f802819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80281006c0 0x7f8028196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80140a3ef0 0x7f80140a63b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 --2- 172.21.15.32:0/1691510353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8028068570 0x7f8028196510 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 >> 172.21.15.32:0/1691510353 conn(0x7f80280fc2b0 msgr2=0x7f80280fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 shutdown_connections 2024-04-23T18:21:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:21:37.668+0000 7f802d537640 1 -- 172.21.15.32:0/1691510353 wait complete. 2024-04-23T18:21:37.672 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a028eb5d', quiesce took 1.1 seconds. Will release after: 30.0 2024-04-23T18:21:38.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:37 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/156484945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:38.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:37 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/156484945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:38.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:37 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/156484945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 370.0 s, 2844.0 tps, lat 25.390 ms stddev 26.374 2024-04-23T18:21:39.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:38 smithi032 ceph-mon[29221]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 MiB/s wr, 570 op/s 2024-04-23T18:21:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:38 smithi040 ceph-mon[32626]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 MiB/s wr, 570 op/s 2024-04-23T18:21:39.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:38 smithi064 ceph-mon[33062]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 MiB/s wr, 570 op/s 2024-04-23T18:21:41.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:40 smithi032 ceph-mon[29221]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-23T18:21:41.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:40 smithi040 ceph-mon[32626]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-23T18:21:41.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:40 smithi064 ceph-mon[33062]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-23T18:21:43.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:42 smithi032 ceph-mon[29221]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.3 MiB/s wr, 607 op/s 2024-04-23T18:21:43.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:42 smithi040 ceph-mon[32626]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.3 MiB/s wr, 607 op/s 2024-04-23T18:21:43.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:42 smithi064 ceph-mon[33062]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.3 MiB/s wr, 607 op/s 2024-04-23T18:21:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 375.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:21:45.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:44 smithi032 ceph-mon[29221]: pgmap v202: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.3 MiB/s wr, 611 op/s 2024-04-23T18:21:45.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:44 smithi040 ceph-mon[32626]: pgmap v202: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.3 MiB/s wr, 611 op/s 2024-04-23T18:21:45.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:44 smithi064 ceph-mon[33062]: pgmap v202: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.3 MiB/s wr, 611 op/s 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 -- 172.21.15.32:0/1295520097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 msgr2=0x7f74a4101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/1295520097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4101fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f749402a410 comp rx=0 tx=0).stop 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 -- 172.21.15.32:0/1295520097 shutdown_connections 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/1295520097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a41061c0 0x7f74a410dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/1295520097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41025b0 0x7f74a4105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/1295520097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4101fe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 -- 172.21.15.32:0/1295520097 >> 172.21.15.32:0/1295520097 conn(0x7f74a4076200 msgr2=0x7f74a4076610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 -- 172.21.15.32:0/1295520097 shutdown_connections 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 -- 172.21.15.32:0/1295520097 wait complete. 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.890+0000 7f74a9ff7640 1 Processor -- start 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 -- start start 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41061c0 0x7f74a4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a419d020 0x7f74a419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4112220 con 0x7f74a4101c00 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41120a0 con 0x7f74a419d020 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a9ff7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41123a0 con 0x7f74a41061c0 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49418/0 (socket says 172.21.15.32:49418) 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 -- 172.21.15.32:0/4278146395 learned_addr learned my addr 172.21.15.32:0/4278146395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a3fff640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a419d020 0x7f74a419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41061c0 0x7f74a4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 -- 172.21.15.32:0/4278146395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41061c0 msgr2=0x7f74a4198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41061c0 0x7f74a4198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 -- 172.21.15.32:0/4278146395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a419d020 msgr2=0x7f74a419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a419d020 0x7f74a419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.891+0000 7f74a37fe640 1 -- 172.21.15.32:0/4278146395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a419db10 con 0x7f74a4101c00 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a37fe640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7494005940 tx=0x7f7494004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a0ff9640 1 -- 172.21.15.32:0/4278146395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749403f030 con 0x7f74a4101c00 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a0ff9640 1 -- 172.21.15.32:0/4278146395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7494003aa0 con 0x7f74a4101c00 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a419dda0 con 0x7f74a4101c00 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a0ff9640 1 -- 172.21.15.32:0/4278146395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494035070 con 0x7f74a4101c00 2024-04-23T18:21:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.892+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a41a5560 con 0x7f74a4101c00 2024-04-23T18:21:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.894+0000 7f74a0ff9640 1 -- 172.21.15.32:0/4278146395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7494042030 con 0x7f74a4101c00 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.894+0000 7f74acde7740 1 Processor -- start 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 -- start start 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecefe60 0x55cdbecfe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed01000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cdbec54970 con 0x55cdbecef540 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cdbebf14a0 con 0x55cdbecefe60 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74acde7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cdbec50930 con 0x55cdbecfec10 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49426/0 (socket says 172.21.15.32:49426) 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 -- 172.21.15.32:0/3952934253 learned_addr learned my addr 172.21.15.32:0/3952934253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a37fe640 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecefe60 0x55cdbecfe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a0ff9640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 0x7f748c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed01000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 0x7f748c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 -- 172.21.15.32:0/3952934253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 msgr2=0x55cdbed01000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed01000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 -- 172.21.15.32:0/3952934253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecefe60 msgr2=0x55cdbecfe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecefe60 0x55cdbecfe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a3fff640 1 -- 172.21.15.32:0/3952934253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdbe967210 con 0x55cdbecef540 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.895+0000 7f74a0ff9640 1 -- 172.21.15.32:0/4278146395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f74940f0f50 con 0x7f74a4101c00 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f74a3fff640 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f749c0149c0 tx=0x7f749c014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68645e56894b3bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f74a3fff640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 0x7f748c0a61d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f749c012760 tx=0x7f749c01e000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f7489ffb640 1 -- 172.21.15.32:0/3952934253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c014050 con 0x55cdbecef540 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f7489ffb640 1 -- 172.21.15.32:0/3952934253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f749c00cd90 con 0x55cdbecef540 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f7489ffb640 1 -- 172.21.15.32:0/3952934253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c013940 con 0x55cdbecef540 2024-04-23T18:21:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f74acde7740 1 -- 172.21.15.32:0/3952934253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 msgr2=0x55cdbecef920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.896+0000 7f74acde7740 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f749c0149c0 tx=0x7f749c014e80 comp rx=0 tx=0).stop 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 -- 172.21.15.32:0/3952934253 shutdown_connections 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed01000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecefe60 0x55cdbecfe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 --2- 172.21.15.32:0/3952934253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecef540 0x55cdbecef920 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 -- 172.21.15.32:0/3952934253 >> 172.21.15.32:0/3952934253 conn(0x55cdbecebb30 msgr2=0x55cdbecebf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 -- 172.21.15.32:0/3952934253 shutdown_connections 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 -- 172.21.15.32:0/3952934253 wait complete. 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 -- start start 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 Processor -- start 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.897+0000 7f74acde7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecef540 0x55cdbecfe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecfec10 0x55cdbecfbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cdbed017e0 con 0x55cdbecefe60 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cdbebef0b0 con 0x55cdbecfec10 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cdbed01960 con 0x55cdbecef540 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 10 client.0 _ll_register_callbacks cb 0x55cdbebdcd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49442/0 (socket says 172.21.15.32:49442) 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 -- 172.21.15.32:0/3263677276 learned_addr learned my addr 172.21.15.32:0/3263677276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a37fe640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecef540 0x55cdbecfe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a3fff640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecfec10 0x55cdbecfbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 -- 172.21.15.32:0/3263677276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecef540 msgr2=0x55cdbecfe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecef540 0x55cdbecfe5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 -- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecfec10 msgr2=0x55cdbecfbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecfec10 0x55cdbecfbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 -- 172.21.15.32:0/3263677276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdbea33620 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7498012700 tx=0x7f7498012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b20ac234fa66bf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f7482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498004260 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7498004400 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.898+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749c00cd90 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f74980035f0 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 10 client.16308 fetch_fsmap learned FSMap version 36 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cdbed0d790 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f749c00cd90 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f7498011b40 con 0x55cdbecefe60 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 10 client.16308 fetch_fsmap finished waiting for FSMap version 36 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 20 client.16308 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed110b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 4 client.16308 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdbec62390 con 0x55cdbed0ecc0 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.899+0000 7f74a37fe640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed110b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.900+0000 7f7482ffd640 10 client.16308 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.900+0000 7f74a37fe640 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed110b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86117edeb4fcdce1 server_cookie=70f76dce44b82550 in_seq=0 out_seq=0 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.900+0000 7f7482ffd640 1 -- 172.21.15.32:0/3263677276 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdbec62390 con 0x55cdbed0ecc0 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.900+0000 7f7482ffd640 10 client.16308 handle_command_reply: tid=0 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 client.16308 shutdown 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 msgr2=0x55cdbecfb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7498012700 tx=0x7f7498012b00 comp rx=0 tx=0).stop 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 shutdown_connections 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed110b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecfec10 0x55cdbecfbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecfb890 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f7482ffd640 0 client.16308 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 --2- 172.21.15.32:0/3263677276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecef540 0x55cdbecfe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 >> 172.21.15.32:0/3263677276 conn(0x55cdbecebb30 msgr2=0x55cdbecf19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 shutdown_connections 2024-04-23T18:21:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 1 -- 172.21.15.32:0/3263677276 wait complete. 2024-04-23T18:21:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.901+0000 7f74acde7740 20 client.16308 trim_cache size 0 max 16384 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 Processor -- start 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 -- start start 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbed0a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecefe60 0x55cdbecfe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41ae200 con 0x55cdbecfec10 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41aba10 con 0x55cdbecef540 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74acde7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41a9880 con 0x55cdbecefe60 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49452/0 (socket says 172.21.15.32:49452) 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a3fff640 1 -- 172.21.15.32:0/775090934 learned_addr learned my addr 172.21.15.32:0/775090934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecefe60 0x55cdbecfe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a3fff640 1 -- 172.21.15.32:0/775090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecefe60 msgr2=0x55cdbecfe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.916+0000 7f74a3fff640 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecefe60 0x55cdbecfe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74a3fff640 1 -- 172.21.15.32:0/775090934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 msgr2=0x55cdbed0a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74a3fff640 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbed0a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74a3fff640 1 -- 172.21.15.32:0/775090934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdbed11420 con 0x55cdbecfec10 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74a3fff640 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c7f0 tx=0x7f749c009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbe561e5f17d6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74897fa640 1 -- 172.21.15.32:0/775090934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c02e020 con 0x55cdbecfec10 2024-04-23T18:21:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74897fa640 1 -- 172.21.15.32:0/775090934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f749c009860 con 0x55cdbecfec10 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74897fa640 1 -- 172.21.15.32:0/775090934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c024040 con 0x55cdbecfec10 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74acde7740 1 -- 172.21.15.32:0/775090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 msgr2=0x55cdbed169a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.917+0000 7f74acde7740 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c7f0 tx=0x7f749c009760 comp rx=0 tx=0).stop 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- 172.21.15.32:0/775090934 shutdown_connections 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecfec10 0x55cdbed169a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecefe60 0x55cdbecfe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- 172.21.15.32:0/775090934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbed0a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- 172.21.15.32:0/775090934 >> 172.21.15.32:0/775090934 conn(0x55cdbecebb30 msgr2=0x55cdbecf6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- 172.21.15.32:0/775090934 shutdown_connections 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- 172.21.15.32:0/775090934 wait complete. 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- start start 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 Processor -- start 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbecf89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed74210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41a9ed0 con 0x55cdbecefe60 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41ab340 con 0x55cdbecef540 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41aa5a0 con 0x55cdbecfec10 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74acde7740 10 client.0 _ll_register_callbacks cb 0x7f7498009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.918+0000 7f74a2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49458/0 (socket says 172.21.15.32:49458) 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 -- 172.21.15.32:0/2967580392 learned_addr learned my addr 172.21.15.32:0/2967580392 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a3fff640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed74210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a37fe640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbecf89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 -- 172.21.15.32:0/2967580392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 msgr2=0x55cdbed74210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed74210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 -- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 msgr2=0x55cdbecf89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbecf89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 -- 172.21.15.32:0/2967580392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdbecfa860 con 0x55cdbecefe60 2024-04-23T18:21:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a37fe640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbecf89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74a2ffd640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7498000c00 tx=0x7f7498009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c717872b96fbb1c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f7482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749802c040 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7498003790 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749c009860 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f74980039f0 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74acde7740 10 client.16314 fetch_fsmap learned FSMap version 36 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.919+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cdbed75730 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f749c009860 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f7498026650 con 0x55cdbecefe60 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 10 client.16314 fetch_fsmap finished waiting for FSMap version 36 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 20 client.16314 populate_metadata read hostname 'smithi032' 2024-04-23T18:21:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed77f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 4 client.16314 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cdbec5c200 con 0x55cdbed0ecc0 2024-04-23T18:21:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74a37fe640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed77f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f7482ffd640 10 client.16314 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.920+0000 7f74a37fe640 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed77f70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feefe6c2f4725501 server_cookie=75472ecdd388b217 in_seq=0 out_seq=0 2024-04-23T18:21:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f7482ffd640 1 -- 172.21.15.32:0/2967580392 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cdbec5c200 con 0x55cdbed0ecc0 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f7482ffd640 10 client.16314 handle_command_reply: tid=0 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 client.16314 shutdown 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 msgr2=0x55cdbecf8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7498000c00 tx=0x7f7498009cb0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 shutdown_connections 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55cdbed0ecc0 0x55cdbed77f70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cdbecfec10 0x55cdbed74210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbecefe60 0x55cdbecf8ef0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 --2- 172.21.15.32:0/2967580392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdbecef540 0x55cdbecf89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f7482ffd640 0 client.16314 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 >> 172.21.15.32:0/2967580392 conn(0x55cdbecebb30 msgr2=0x55cdbecea420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 shutdown_connections 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 1 -- 172.21.15.32:0/2967580392 wait complete. 2024-04-23T18:21:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74acde7740 20 client.16314 trim_cache size 0 max 16384 2024-04-23T18:21:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:21:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:21:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:21:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 msgr2=0x7f748c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 0x7f748c0a61d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f749c012760 tx=0x7f749c01e000 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 msgr2=0x7f74a4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7494005940 tx=0x7f7494004270 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 shutdown_connections 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a419d020 0x7f74a419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74a41061c0 0x7f74a4198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f748c0a3d10 0x7f748c0a61d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 --2- 172.21.15.32:0/4278146395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a4101c00 0x7f74a4198760 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.942+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 >> 172.21.15.32:0/4278146395 conn(0x7f74a4076200 msgr2=0x7f74a40fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.943+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 shutdown_connections 2024-04-23T18:21:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:21:46.943+0000 7f74a9ff7640 1 -- 172.21.15.32:0/4278146395 wait complete. 2024-04-23T18:21:47.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:46 smithi032 ceph-mon[29221]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-04-23T18:21:47.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:46 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:46 smithi040 ceph-mon[32626]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-04-23T18:21:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:46 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:47.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:46 smithi064 ceph-mon[33062]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-04-23T18:21:47.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:46 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:48.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 380.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:21:49.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:48 smithi032 ceph-mon[29221]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.2 MiB/s wr, 325 op/s 2024-04-23T18:21:49.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:48 smithi040 ceph-mon[32626]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.2 MiB/s wr, 325 op/s 2024-04-23T18:21:49.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:48 smithi064 ceph-mon[33062]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.2 MiB/s wr, 325 op/s 2024-04-23T18:21:51.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:50 smithi032 ceph-mon[29221]: pgmap v205: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 4.3 MiB/s wr, 335 op/s 2024-04-23T18:21:51.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:50 smithi040 ceph-mon[32626]: pgmap v205: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 4.3 MiB/s wr, 335 op/s 2024-04-23T18:21:51.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:50 smithi064 ceph-mon[33062]: pgmap v205: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 4.3 MiB/s wr, 335 op/s 2024-04-23T18:21:53.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:52 smithi032 ceph-mon[29221]: pgmap v206: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 14 op/s 2024-04-23T18:21:53.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:52 smithi040 ceph-mon[32626]: pgmap v206: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 14 op/s 2024-04-23T18:21:53.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:52 smithi064 ceph-mon[33062]: pgmap v206: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 14 op/s 2024-04-23T18:21:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 385.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:21:55.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:54 smithi032 ceph-mon[29221]: pgmap v207: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-04-23T18:21:55.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:54 smithi040 ceph-mon[32626]: pgmap v207: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-04-23T18:21:55.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:54 smithi064 ceph-mon[33062]: pgmap v207: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-04-23T18:21:57.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:56 smithi032 ceph-mon[29221]: pgmap v208: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:21:57.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:56 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:57.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:56 smithi040 ceph-mon[32626]: pgmap v208: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:21:57.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:56 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:57.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:56 smithi064 ceph-mon[33062]: pgmap v208: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:21:57.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:56 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 390.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:21:59.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:21:58 smithi032 ceph-mon[29221]: pgmap v209: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:21:59.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:21:58 smithi040 ceph-mon[32626]: pgmap v209: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:21:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:21:58 smithi064 ceph-mon[33062]: pgmap v209: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-23T18:22:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.043+0000 7f0196b44640 1 -- 172.21.15.32:0/2822571629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 msgr2=0x7f01901025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.043+0000 7f0196b44640 1 --2- 172.21.15.32:0/2822571629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f01901025a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f018c009900 tx=0x7f018c02a410 comp rx=0 tx=0).stop 2024-04-23T18:22:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 -- 172.21.15.32:0/2822571629 shutdown_connections 2024-04-23T18:22:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 --2- 172.21.15.32:0/2822571629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0190107290 0x7f019010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 --2- 172.21.15.32:0/2822571629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190102b70 0x7f0190106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 --2- 172.21.15.32:0/2822571629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f01901025a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 -- 172.21.15.32:0/2822571629 >> 172.21.15.32:0/2822571629 conn(0x7f01900fda40 msgr2=0x7f01900ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 -- 172.21.15.32:0/2822571629 shutdown_connections 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 -- 172.21.15.32:0/2822571629 wait complete. 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 Processor -- start 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 -- start start 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.044+0000 7f0196b44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190107290 0x7f019019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0196b44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01901985b0 0x7f0190198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0196b44640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01901121d0 con 0x7f01901021c0 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0196b44640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0190112050 con 0x7f01901985b0 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0196b44640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0190112350 con 0x7f0190107290 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33122/0 (socket says 172.21.15.32:33122) 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 -- 172.21.15.32:0/3897736421 learned_addr learned my addr 172.21.15.32:0/3897736421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0187fff640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190107290 0x7f019019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01950e1640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01901985b0 0x7f0190198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 -- 172.21.15.32:0/3897736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190107290 msgr2=0x7f019019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190107290 0x7f019019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 -- 172.21.15.32:0/3897736421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01901985b0 msgr2=0x7f0190198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01901985b0 0x7f0190198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 -- 172.21.15.32:0/3897736421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01901990d0 con 0x7f01901021c0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f01948e0640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f018c005940 tx=0x7f018c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.045+0000 7f0185ffb640 1 -- 172.21.15.32:0/3897736421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018c03f030 con 0x7f01901021c0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.046+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0190199360 con 0x7f01901021c0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.046+0000 7f0185ffb640 1 -- 172.21.15.32:0/3897736421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f018c003cb0 con 0x7f01901021c0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.046+0000 7f0185ffb640 1 -- 172.21.15.32:0/3897736421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018c035070 con 0x7f01901021c0 2024-04-23T18:22:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.046+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01901a5550 con 0x7f01901021c0 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.048+0000 7f0185ffb640 1 -- 172.21.15.32:0/3897736421 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f018c042030 con 0x7f01901021c0 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.048+0000 7f019985e740 1 Processor -- start 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.048+0000 7f019985e740 1 -- start start 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f0185ffb640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 0x7f01780a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c28c10 0x561d57c2b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561d57b7e970 con 0x561d57c19520 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d57b1b4a0 con 0x561d57c28c10 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f019985e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561d57b7a930 con 0x561d57c19e40 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f0187fff640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 0x7f01780a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33124/0 (socket says 172.21.15.32:33124) 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 -- 172.21.15.32:0/986890427 learned_addr learned my addr 172.21.15.32:0/986890427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01948e0640 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f0185ffb640 1 -- 172.21.15.32:0/3897736421 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f018c0f0f80 con 0x7f01901021c0 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c28c10 0x561d57c2b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 -- 172.21.15.32:0/986890427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 msgr2=0x561d57c286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c286d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 -- 172.21.15.32:0/986890427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c28c10 msgr2=0x561d57c2b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c28c10 0x561d57c2b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.049+0000 7f01950e1640 1 -- 172.21.15.32:0/986890427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d57891210 con 0x561d57c19520 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f0187fff640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 0x7f01780a61d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f01900fded0 tx=0x7f018000b410 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f01950e1640 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f018800af70 tx=0x7f018800a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cce685d5329ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f0176ffd640 1 -- 172.21.15.32:0/986890427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188019030 con 0x561d57c19520 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f0176ffd640 1 -- 172.21.15.32:0/986890427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0188013e30 con 0x561d57c19520 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f0176ffd640 1 -- 172.21.15.32:0/986890427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188003bd0 con 0x561d57c19520 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 -- 172.21.15.32:0/986890427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 msgr2=0x561d57c19900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f018800af70 tx=0x7f018800a980 comp rx=0 tx=0).stop 2024-04-23T18:22:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 -- 172.21.15.32:0/986890427 shutdown_connections 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c28c10 0x561d57c2b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c286d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 --2- 172.21.15.32:0/986890427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c19900 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.050+0000 7f019985e740 1 -- 172.21.15.32:0/986890427 >> 172.21.15.32:0/986890427 conn(0x561d57c15b30 msgr2=0x561d57c15f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- 172.21.15.32:0/986890427 shutdown_connections 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- 172.21.15.32:0/986890427 wait complete. 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- start start 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 Processor -- start 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19e40 0x561d57c25790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c25cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561d57c2b840 con 0x561d57c19520 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d57b190b0 con 0x561d57c19e40 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561d57c2b9c0 con 0x561d57c28c10 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.051+0000 7f019985e740 10 client.0 _ll_register_callbacks cb 0x561d57b06d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33138/0 (socket says 172.21.15.32:33138) 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 -- 172.21.15.32:0/1568627473 learned_addr learned my addr 172.21.15.32:0/1568627473 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01950e1640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c25cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f0187fff640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19e40 0x561d57c25790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 -- 172.21.15.32:0/1568627473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 msgr2=0x561d57c25cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c25cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 -- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19e40 msgr2=0x561d57c25790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19e40 0x561d57c25790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 -- 172.21.15.32:0/1568627473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5795d620 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f01948e0640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f018c003fd0 tx=0x7f018c047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42adcb96356be65a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f016ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018c049070 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f018c0b9690 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0188013e30 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f018c005d50 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f019985e740 10 client.16326 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.052+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561d57c3bb20 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0188013e30 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f018c0db140 con 0x561d57c19520 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 10 client.16326 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 10 client.16326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 20 client.16326 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57c3cfd0 0x561d57c3f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 4 client.16326 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d57b8c390 con 0x561d57c3cfd0 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.053+0000 7f0187fff640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57c3cfd0 0x561d57c3f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f0187fff640 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57c3cfd0 0x561d57c3f420 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7213f9a9b289a4c1 server_cookie=3c73f1fb4e586682 in_seq=0 out_seq=0 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f016ffff640 10 client.16326 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f016ffff640 1 -- 172.21.15.32:0/1568627473 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d57b8c390 con 0x561d57c3cfd0 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f016ffff640 10 client.16326 handle_command_reply: tid=0 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 client.16326 shutdown 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 msgr2=0x561d57c28240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f018c003fd0 tx=0x7f018c047000 comp rx=0 tx=0).stop 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 shutdown_connections 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57c3cfd0 0x561d57c3f420 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c25cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19e40 0x561d57c25790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f016ffff640 0 client.16326 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 --2- 172.21.15.32:0/1568627473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19520 0x561d57c28240 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.054+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 >> 172.21.15.32:0/1568627473 conn(0x561d57c15b30 msgr2=0x561d57c1b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.055+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 shutdown_connections 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.055+0000 7f019985e740 1 -- 172.21.15.32:0/1568627473 wait complete. 2024-04-23T18:22:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.055+0000 7f019985e740 20 client.16326 trim_cache size 0 max 16384 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 Processor -- start 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 -- start start 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c44d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c22a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01901ae1c0 con 0x561d57c19e40 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01901aba00 con 0x561d57c19520 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f019985e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01901a9850 con 0x561d57c28c10 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33152/0 (socket says 172.21.15.32:33152) 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 -- 172.21.15.32:0/3449962232 learned_addr learned my addr 172.21.15.32:0/3449962232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f01948e0640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c44d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f01950e1640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c22a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 -- 172.21.15.32:0/3449962232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 msgr2=0x561d57c22a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c22a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 -- 172.21.15.32:0/3449962232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 msgr2=0x561d57c44d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c44d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f0187fff640 1 -- 172.21.15.32:0/3449962232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d57c15960 con 0x561d57c19e40 2024-04-23T18:22:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.070+0000 7f01950e1640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c22a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f0187fff640 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0180005d00 tx=0x7f0180013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53a00e2713be63d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f01767fc640 1 -- 172.21.15.32:0/3449962232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180011e60 con 0x561d57c19e40 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f01767fc640 1 -- 172.21.15.32:0/3449962232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f018002e070 con 0x561d57c19e40 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f01767fc640 1 -- 172.21.15.32:0/3449962232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018001fbd0 con 0x561d57c19e40 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 -- 172.21.15.32:0/3449962232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 msgr2=0x561d57c45240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0180005d00 tx=0x7f0180013000 comp rx=0 tx=0).stop 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 -- 172.21.15.32:0/3449962232 shutdown_connections 2024-04-23T18:22:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c28c10 0x561d57c22a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c19e40 0x561d57c45240 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 --2- 172.21.15.32:0/3449962232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c44d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 -- 172.21.15.32:0/3449962232 >> 172.21.15.32:0/3449962232 conn(0x561d57c15b30 msgr2=0x561d57c14cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.071+0000 7f019985e740 1 -- 172.21.15.32:0/3449962232 shutdown_connections 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 -- 172.21.15.32:0/3449962232 wait complete. 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 -- start start 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 Processor -- start 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c37500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c37a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01901a9ec0 con 0x561d57c28c10 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01901ab330 con 0x561d57c19520 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01901aa590 con 0x561d57c19e40 2024-04-23T18:22:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f019985e740 10 client.0 _ll_register_callbacks cb 0x7f018c0b92c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f01950e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f01950e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33162/0 (socket says 172.21.15.32:33162) 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f01950e1640 1 -- 172.21.15.32:0/3921763411 learned_addr learned my addr 172.21.15.32:0/3921763411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.072+0000 7f0187fff640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c37a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01948e0640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c37500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 -- 172.21.15.32:0/3921763411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 msgr2=0x561d57c37a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c37a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 -- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 msgr2=0x561d57c37500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c37500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 -- 172.21.15.32:0/3921763411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d57c1ab30 con 0x561d57c28c10 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f01950e1640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f01880117b0 tx=0x7f0188011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cce12fc2787938 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f016ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188019040 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0188013250 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f018002e070 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0188013450 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f019985e740 10 client.16332 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.073+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561d57ca36e0 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f018002e070 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f0188005730 con 0x561d57c28c10 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 10 client.16332 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 20 client.16332 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57ca3a70 0x561d57ca5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 4 client.16332 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d57b86200 con 0x561d57ca3a70 2024-04-23T18:22:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f01948e0640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57ca3a70 0x561d57ca5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f016ffff640 10 client.16332 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.074+0000 7f01948e0640 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57ca3a70 0x561d57ca5e60 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a70a41cc88cb158 server_cookie=3e0af9ea8d1041df in_seq=0 out_seq=0 2024-04-23T18:22:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f016ffff640 1 -- 172.21.15.32:0/3921763411 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d57b86200 con 0x561d57ca3a70 2024-04-23T18:22:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f016ffff640 10 client.16332 handle_command_reply: tid=0 2024-04-23T18:22:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 client.16332 shutdown 2024-04-23T18:22:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 msgr2=0x561d57c37f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f01880117b0 tx=0x7f0188011d20 comp rx=0 tx=0).stop 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 shutdown_connections 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561d57ca3a70 0x561d57ca5e60 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561d57c28c10 0x561d57c37f80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561d57c19e40 0x561d57c37a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f016ffff640 0 client.16332 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 --2- 172.21.15.32:0/3921763411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d57c19520 0x561d57c37500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 >> 172.21.15.32:0/3921763411 conn(0x561d57c15b30 msgr2=0x561d57c24ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 shutdown_connections 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.088+0000 7f019985e740 1 -- 172.21.15.32:0/3921763411 wait complete. 2024-04-23T18:22:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f019985e740 20 client.16332 trim_cache size 0 max 16384 2024-04-23T18:22:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:22:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:22:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:22:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 msgr2=0x7f01780a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 0x7f01780a61d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f01900fded0 tx=0x7f018000b410 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 msgr2=0x7f019019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f018c005940 tx=0x7f018c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 shutdown_connections 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01901985b0 0x7f0190198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0190107290 0x7f019019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f01780a3d10 0x7f01780a61d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 --2- 172.21.15.32:0/3897736421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01901021c0 0x7f019019e400 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 >> 172.21.15.32:0/3897736421 conn(0x7f01900fda40 msgr2=0x7f01900fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 shutdown_connections 2024-04-23T18:22:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:00.089+0000 7f0196b44640 1 -- 172.21.15.32:0/3897736421 wait complete. 2024-04-23T18:22:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:00 smithi032 ceph-mon[29221]: pgmap v210: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-04-23T18:22:01.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:00 smithi040 ceph-mon[32626]: pgmap v210: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-04-23T18:22:01.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:00 smithi064 ceph-mon[33062]: pgmap v210: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-04-23T18:22:03.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:02 smithi032 ceph-mon[29221]: pgmap v211: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-04-23T18:22:03.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:02 smithi040 ceph-mon[32626]: pgmap v211: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-04-23T18:22:03.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:02 smithi064 ceph-mon[33062]: pgmap v211: 97 pgs: 97 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-04-23T18:22:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 395.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:05.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:04 smithi032 ceph-mon[29221]: pgmap v212: 97 pgs: 97 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 324 op/s 2024-04-23T18:22:05.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:04 smithi040 ceph-mon[32626]: pgmap v212: 97 pgs: 97 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 324 op/s 2024-04-23T18:22:05.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:04 smithi064 ceph-mon[33062]: pgmap v212: 97 pgs: 97 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 324 op/s 2024-04-23T18:22:07.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:06 smithi032 ceph-mon[29221]: pgmap v213: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 487 op/s 2024-04-23T18:22:07.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:06 smithi032 ceph-mon[29221]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:06 smithi040 ceph-mon[32626]: pgmap v213: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 487 op/s 2024-04-23T18:22:07.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:06 smithi040 ceph-mon[32626]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:06 smithi064 ceph-mon[33062]: pgmap v213: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 487 op/s 2024-04-23T18:22:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:06 smithi064 ceph-mon[33062]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:07.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.784+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/3640459558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 msgr2=0x7ff3e81072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.784+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/3640459558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81072b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009900 tx=0x7ff3d002a410 comp rx=0 tx=0).stop 2024-04-23T18:22:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/3640459558 shutdown_connections 2024-04-23T18:22:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/3640459558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 0x7ff3e810dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/3640459558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 0x7ff3e8101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/3640459558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81072b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/3640459558 >> 172.21.15.32:0/3640459558 conn(0x7ff3e8077c40 msgr2=0x7ff3e8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/3640459558 shutdown_connections 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/3640459558 wait complete. 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 Processor -- start 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 -- start start 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 0x7ff3e8199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 0x7ff3e8199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.785+0000 7ff3ef0fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ef0fd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8111080 con 0x7ff3e8106ed0 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ef0fd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8110f00 con 0x7ff3e81010a0 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ef0fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8111200 con 0x7ff3e8101a40 2024-04-23T18:22:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35868/0 (socket says 172.21.15.32:35868) 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 -- 172.21.15.32:0/2610772494 learned_addr learned my addr 172.21.15.32:0/2610772494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3e7fff640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 0x7ff3e8199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ece99640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 0x7ff3e8199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 -- 172.21.15.32:0/2610772494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 msgr2=0x7ff3e8199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 0x7ff3e8199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 -- 172.21.15.32:0/2610772494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 msgr2=0x7ff3e8199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 0x7ff3e8199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 -- 172.21.15.32:0/2610772494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e81a4ae0 con 0x7ff3e8106ed0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ed69a640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000eaa0 tx=0x7ff3e000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004d20 con 0x7ff3e8106ed0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e0005020 con 0x7ff3e8106ed0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00116b0 con 0x7ff3e8106ed0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.786+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e81a4cb0 con 0x7ff3e8106ed0 2024-04-23T18:22:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.787+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e81a5180 con 0x7ff3e8106ed0 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.788+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e8068e00 con 0x7ff3e8106ed0 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.788+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff3e00051d0 con 0x7ff3e8106ed0 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.789+0000 7ff3e5ffb640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 0x7ff3d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.789+0000 7ff3ece99640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 0x7ff3d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.789+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7ff3e00d0930 con 0x7ff3e8106ed0 2024-04-23T18:22:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.790+0000 7ff3ece99640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 0x7ff3d40a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0031440 tx=0x7ff3d0005b50 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:07.794+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3e0016030 con 0x7ff3e8106ed0 2024-04-23T18:22:08.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.018+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3e8109310 con 0x7ff3e8106ed0 2024-04-23T18:22:08.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.019+0000 7ff3e5ffb640 1 -- 172.21.15.32:0/2610772494 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11855 (secure 0 0 0) 0x7ff3e009cfd0 con 0x7ff3e8106ed0 2024-04-23T18:22:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:22:08.014 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:21:55.867165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:22:08.014 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2024-04-23T18:22:08.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 msgr2=0x7ff3d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 0x7ff3d40a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0031440 tx=0x7ff3d0005b50 comp rx=0 tx=0).stop 2024-04-23T18:22:08.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 msgr2=0x7ff3e81a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000eaa0 tx=0x7ff3e000eea0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 shutdown_connections 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff3d40a3d20 0x7ff3d40a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e8106ed0 0x7ff3e81a45a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3e8101a40 0x7ff3e8199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 --2- 172.21.15.32:0/2610772494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3e81010a0 0x7ff3e8199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 >> 172.21.15.32:0/2610772494 conn(0x7ff3e8077c40 msgr2=0x7ff3e80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 shutdown_connections 2024-04-23T18:22:08.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.021+0000 7ff3ef0fd640 1 -- 172.21.15.32:0/2610772494 wait complete. 2024-04-23T18:22:08.027 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id a028eb5d --release --await' 2024-04-23T18:22:08.027 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id a028eb5d --release --await 2024-04-23T18:22:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 -- 172.21.15.32:0/1127803982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1020103b70 msgr2=0x7f102010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 --2- 172.21.15.32:0/1127803982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1020103b70 0x7f102010bfc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1014009900 tx=0x7f101402a410 comp rx=0 tx=0).stop 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 -- 172.21.15.32:0/1127803982 shutdown_connections 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 --2- 172.21.15.32:0/1127803982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102010c6d0 0x7f102010eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 --2- 172.21.15.32:0/1127803982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1020103b70 0x7f102010bfc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 --2- 172.21.15.32:0/1127803982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10201031c0 0x7f10201035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 -- 172.21.15.32:0/1127803982 >> 172.21.15.32:0/1127803982 conn(0x7f1020077f70 msgr2=0x7f1020078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 -- 172.21.15.32:0/1127803982 shutdown_connections 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.142+0000 7f1025eef640 1 -- 172.21.15.32:0/1127803982 wait complete. 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 Processor -- start 2024-04-23T18:22:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 -- start start 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f102010c6d0 0x7f102019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020198800 0x7f1020198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1020112350 con 0x7f10201031c0 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10201121d0 con 0x7f1020198800 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f1025eef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10201124d0 con 0x7f102010c6d0 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35886/0 (socket says 172.21.15.32:35886) 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 -- 172.21.15.32:0/4134196986 learned_addr learned my addr 172.21.15.32:0/4134196986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101effd640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f102010c6d0 0x7f102019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101ffff640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020198800 0x7f1020198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 -- 172.21.15.32:0/4134196986 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f102010c6d0 msgr2=0x7f102019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f102010c6d0 0x7f102019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 -- 172.21.15.32:0/4134196986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020198800 msgr2=0x7f1020198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.143+0000 7f101f7fe640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020198800 0x7f1020198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f101f7fe640 1 -- 172.21.15.32:0/4134196986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1020199390 con 0x7f10201031c0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f101f7fe640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f101000d950 tx=0x7f101000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f101cff9640 1 -- 172.21.15.32:0/4134196986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010017070 con 0x7f10201031c0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1020199680 con 0x7f10201031c0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f101cff9640 1 -- 172.21.15.32:0/4134196986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1010009c30 con 0x7f10201031c0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f101cff9640 1 -- 172.21.15.32:0/4134196986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101000f620 con 0x7f10201031c0 2024-04-23T18:22:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.144+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10201a5830 con 0x7f10201031c0 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101cff9640 1 -- 172.21.15.32:0/4134196986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1010010460 con 0x7f10201031c0 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 Processor -- start 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 -- start start 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1c490 0x5592b7e1c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e29870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5592b7d816d0 con 0x5592b7e1cdb0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5592b7d1e1a0 con 0x5592b7e27480 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1028c07740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5592b7d7d690 con 0x5592b7e1c490 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101cff9640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 0x7f10080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35896/0 (socket says 172.21.15.32:35896) 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 -- 172.21.15.32:0/1727120568 learned_addr learned my addr 172.21.15.32:0/1727120568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101effd640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e29870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101ffff640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1c490 0x5592b7e1c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101ffff640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 0x7f10080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101cff9640 1 -- 172.21.15.32:0/4134196986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f10100d03d0 con 0x7f10201031c0 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 -- 172.21.15.32:0/1727120568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1c490 msgr2=0x5592b7e1c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1c490 0x5592b7e1c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 -- 172.21.15.32:0/1727120568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 msgr2=0x5592b7e29870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e29870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 -- 172.21.15.32:0/1727120568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b7d285b0 con 0x5592b7e1cdb0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101effd640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e29870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101f7fe640 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f10100cda20 tx=0x7f101009c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec9c5e1669761fb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f101ffff640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 0x7f10080a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1018009fd0 tx=0x7f1018009600 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.147+0000 7f1005ffb640 1 -- 172.21.15.32:0/1727120568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010020030 con 0x5592b7e1cdb0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1005ffb640 1 -- 172.21.15.32:0/1727120568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10100a06e0 con 0x5592b7e1cdb0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1005ffb640 1 -- 172.21.15.32:0/1727120568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10100a4b10 con 0x5592b7e1cdb0 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 -- 172.21.15.32:0/1727120568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 msgr2=0x5592b7e26f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f10100cda20 tx=0x7f101009c050 comp rx=0 tx=0).stop 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 -- 172.21.15.32:0/1727120568 shutdown_connections 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e29870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e26f40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 --2- 172.21.15.32:0/1727120568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1c490 0x5592b7e1c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 -- 172.21.15.32:0/1727120568 >> 172.21.15.32:0/1727120568 conn(0x5592b7e187b0 msgr2=0x5592b7e18b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 -- 172.21.15.32:0/1727120568 shutdown_connections 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.148+0000 7f1028c07740 1 -- 172.21.15.32:0/1727120568 wait complete. 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 -- start start 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 Processor -- start 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e26e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e27480 0x5592b7e24790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5592b7e2a250 con 0x5592b7e1cdb0 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5592b7d1bb20 con 0x5592b7e1c490 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5592b7e2a3d0 con 0x5592b7e27480 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f1028c07740 10 client.0 _ll_register_callbacks cb 0x5592b7d1b530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35908/0 (socket says 172.21.15.32:35908) 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 -- 172.21.15.32:0/3596679241 learned_addr learned my addr 172.21.15.32:0/3596679241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101ffff640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e27480 0x5592b7e24790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e26e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 -- 172.21.15.32:0/3596679241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e27480 msgr2=0x5592b7e24790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e27480 0x5592b7e24790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 -- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 msgr2=0x5592b7e26e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e26e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.149+0000 7f101effd640 1 -- 172.21.15.32:0/3596679241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b7a93920 con 0x5592b7e1cdb0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e26e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f101effd640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f102006a2d0 tx=0x7f10140314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124d0931b5f9513b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014035040 con 0x5592b7e1cdb0 2024-04-23T18:22:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f1014031810 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10100a06e0 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1014004040 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 10 client.16350 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5592b7b5fd30 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10100a06e0 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f1014004850 con 0x5592b7e1cdb0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 10 client.16350 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 10 client.16350 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 20 client.16350 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e374c0 0x5592b7e398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 4 client.16350 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.150+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592b7d3f500 con 0x5592b7e374c0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.151+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e374c0 0x5592b7e398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.151+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e374c0 0x5592b7e398f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc7325b9e5b0b70 server_cookie=4dcdabf43c72e6ef in_seq=0 out_seq=0 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.151+0000 7f0ffeffd640 10 client.16350 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3596679241 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592b7d3f500 con 0x5592b7e374c0 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f0ffeffd640 10 client.16350 handle_command_reply: tid=0 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 client.16350 shutdown 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 msgr2=0x5592b7e24180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f102006a2d0 tx=0x7f10140314a0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 shutdown_connections 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e374c0 0x5592b7e398f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e27480 0x5592b7e24790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1cdb0 0x5592b7e24180 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f0ffeffd640 0 client.16350 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 --2- 172.21.15.32:0/3596679241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e26e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 >> 172.21.15.32:0/3596679241 conn(0x5592b7e187b0 msgr2=0x5592b7e1e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 shutdown_connections 2024-04-23T18:22:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 1 -- 172.21.15.32:0/3596679241 wait complete. 2024-04-23T18:22:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.152+0000 7f1028c07740 20 client.16350 trim_cache size 0 max 16384 2024-04-23T18:22:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.169+0000 7f1028c07740 1 Processor -- start 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.169+0000 7f1028c07740 1 -- start start 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e1ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10201b0570 con 0x5592b7e27480 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10201addb0 con 0x5592b7e1c490 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f1028c07740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10201abc20 con 0x5592b7e1cdb0 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35922/0 (socket says 172.21.15.32:35922) 2024-04-23T18:22:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 -- 172.21.15.32:0/2564869933 learned_addr learned my addr 172.21.15.32:0/2564869933 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101effd640 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 -- 172.21.15.32:0/2564869933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 msgr2=0x5592b7e269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 -- 172.21.15.32:0/2564869933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 msgr2=0x5592b7e1ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e1ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 -- 172.21.15.32:0/2564869933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b7e1cbe0 con 0x5592b7e27480 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f101ffff640 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f10180129f0 tx=0x7f101800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8210aa7821d6b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f10057fa640 1 -- 172.21.15.32:0/2564869933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101800ae50 con 0x5592b7e27480 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f10057fa640 1 -- 172.21.15.32:0/2564869933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101801e970 con 0x5592b7e27480 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.170+0000 7f10057fa640 1 -- 172.21.15.32:0/2564869933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018005420 con 0x5592b7e27480 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- 172.21.15.32:0/2564869933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 msgr2=0x5592b7e3f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f10180129f0 tx=0x7f101800c000 comp rx=0 tx=0).stop 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- 172.21.15.32:0/2564869933 shutdown_connections 2024-04-23T18:22:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e27480 0x5592b7e3f1c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 --2- 172.21.15.32:0/2564869933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e1c490 0x5592b7e1ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- 172.21.15.32:0/2564869933 >> 172.21.15.32:0/2564869933 conn(0x5592b7e187b0 msgr2=0x5592b7e179d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- 172.21.15.32:0/2564869933 shutdown_connections 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- 172.21.15.32:0/2564869933 wait complete. 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.171+0000 7f1028c07740 1 -- start start 2024-04-23T18:22:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 Processor -- start 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e2fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e203f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10201ac270 con 0x5592b7e1c490 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10201ad6e0 con 0x5592b7e27480 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10201ac940 con 0x5592b7e1cdb0 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 10 client.0 _ll_register_callbacks cb 0x5592b7d1b530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35932/0 (socket says 172.21.15.32:35932) 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 -- 172.21.15.32:0/3819811753 learned_addr learned my addr 172.21.15.32:0/3819811753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101ffff640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e203f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101effd640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e2fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 -- 172.21.15.32:0/3819811753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 msgr2=0x5592b7e2fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e2fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 -- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 msgr2=0x5592b7e203f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e203f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 -- 172.21.15.32:0/3819811753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b7e20a30 con 0x5592b7e1c490 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101effd640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f101f7fe640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1010098840 tx=0x7f101000b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb0197c05f3deb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f0ffeffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010020040 con 0x5592b7e1c490 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f10100cdd60 con 0x5592b7e1c490 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.172+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f101801e970 con 0x5592b7e1c490 2024-04-23T18:22:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f10100cdf60 con 0x5592b7e1c490 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 10 client.16356 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5592b7e21d70 con 0x5592b7e1c490 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101801e970 con 0x5592b7e1c490 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f101009b830 con 0x5592b7e1c490 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 10 client.16356 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 10 client.16356 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 20 client.16356 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e9ea40 0x5592b7ea0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 4 client.16356 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a028eb5d", "release": true, "await": true}] 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a028eb5d", "release": true, "await": true}) -- 0x5592b7d8f0f0 con 0x5592b7e9ea40 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f101effd640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e9ea40 0x5592b7ea0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f0ffeffd640 10 client.16356 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.173+0000 7f101effd640 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e9ea40 0x5592b7ea0e30 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=260c6d205b6f9593 server_cookie=a83f19a8ec63299 in_seq=0 out_seq=0 2024-04-23T18:22:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.184+0000 7f0ffeffd640 1 -- 172.21.15.32:0/3819811753 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5592b7d8f0f0 con 0x5592b7e9ea40 2024-04-23T18:22:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.184+0000 7f0ffeffd640 10 client.16356 handle_command_reply: tid=0 2024-04-23T18:22:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.184+0000 7f1028c07740 1 client.16356 shutdown 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 msgr2=0x5592b7e2f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1010098840 tx=0x7f101000b530 comp rx=0 tx=0).stop 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 shutdown_connections 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5592b7e9ea40 0x5592b7ea0e30 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592b7e27480 0x5592b7e203f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b7e1cdb0 0x5592b7e2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f0ffeffd640 0 client.16356 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 --2- 172.21.15.32:0/3819811753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592b7e1c490 0x5592b7e2f620 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 >> 172.21.15.32:0/3819811753 conn(0x5592b7e187b0 msgr2=0x5592b7e1dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 shutdown_connections 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 1 -- 172.21.15.32:0/3819811753 wait complete. 2024-04-23T18:22:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1028c07740 20 client.16356 trim_cache size 0 max 16384 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 37, 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 24, 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "a028eb5d": { 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 24, 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:22:08.181 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:08.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:08.183 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:08.183 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:22:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 msgr2=0x7f10080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 0x7f10080a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1018009fd0 tx=0x7f1018009600 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.185+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 msgr2=0x7f102019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f101000d950 tx=0x7f101000de10 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 shutdown_connections 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020198800 0x7f1020198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f102010c6d0 0x7f102019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f10080a3d20 0x7f10080a61e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 --2- 172.21.15.32:0/4134196986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10201031c0 0x7f102019e690 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 >> 172.21.15.32:0/4134196986 conn(0x7f1020077f70 msgr2=0x7f10200ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 shutdown_connections 2024-04-23T18:22:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.186+0000 7f1025eef640 1 -- 172.21.15.32:0/4134196986 wait complete. 2024-04-23T18:22:08.194 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a028eb5d', seconds elapsed: 31.6 2024-04-23T18:22:08.194 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:22:08.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 -- 172.21.15.32:0/1412546668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c108f50 msgr2=0x7fb52c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1412546668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c108f50 0x7fb52c109330 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009930 tx=0x7fb51c02a420 comp rx=0 tx=0).stop 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb52a59c640 1 -- 172.21.15.32:0/1412546668 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c0318a0 con 0x7fb52c108f50 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 -- 172.21.15.32:0/1412546668 shutdown_connections 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1412546668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c1150a0 0x7fb52c117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1412546668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c109870 0x7fb52c114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1412546668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c108f50 0x7fb52c109330 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 -- 172.21.15.32:0/1412546668 >> 172.21.15.32:0/1412546668 conn(0x7fb52c06e2e0 msgr2=0x7fb52c06e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 -- 172.21.15.32:0/1412546668 shutdown_connections 2024-04-23T18:22:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.348+0000 7fb5311ad640 1 -- 172.21.15.32:0/1412546668 wait complete. 2024-04-23T18:22:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.349+0000 7fb5311ad640 1 Processor -- start 2024-04-23T18:22:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 -- start start 2024-04-23T18:22:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c108f50 0x7fb52c1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c1150a0 0x7fb52c1a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c11aa40 con 0x7fb52c1150a0 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c11a8c0 con 0x7fb52c108f50 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb5311ad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c11abc0 con 0x7fb52c109870 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb523fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb52b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c1150a0 0x7fb52c1a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb523fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36160/0 (socket says 172.21.15.32:36160) 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.350+0000 7fb523fff640 1 -- 172.21.15.32:0/1456960261 learned_addr learned my addr 172.21.15.32:0/1456960261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb52ad9d640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c108f50 0x7fb52c1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 -- 172.21.15.32:0/1456960261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c108f50 msgr2=0x7fb52c1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c108f50 0x7fb52c1a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 -- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c1150a0 msgr2=0x7fb52c1a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c1150a0 0x7fb52c1a5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 -- 172.21.15.32:0/1456960261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c1a5f70 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.351+0000 7fb523fff640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb51400e9f0 tx=0x7fb51400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.352+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51400cd80 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.352+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5140108e0 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.352+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514018980 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.352+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c1a6260 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.352+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c10a8f0 con 0x7fb52c109870 2024-04-23T18:22:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.353+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c068e00 con 0x7fb52c109870 2024-04-23T18:22:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.355+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb514010430 con 0x7fb52c109870 2024-04-23T18:22:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.356+0000 7fb528d99640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 0x7fb50c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.356+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7fb5140d0b20 con 0x7fb52c109870 2024-04-23T18:22:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.356+0000 7fb52ad9d640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 0x7fb50c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.356+0000 7fb52ad9d640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 0x7fb50c0a6360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0096c0 tx=0x7fb51c031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.361+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb51409d180 con 0x7fb52c109870 2024-04-23T18:22:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.793+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb52c1a22d0 con 0x7fb52c109870 2024-04-23T18:22:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.794+0000 7fb528d99640 1 -- 172.21.15.32:0/1456960261 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11855 (secure 0 0 0) 0x7fb51409ca80 con 0x7fb52c109870 2024-04-23T18:22:08.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:22:08.787 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:21:55.867165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:22:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2024-04-23T18:22:08.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 msgr2=0x7fb50c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 0x7fb50c0a6360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0096c0 tx=0x7fb51c031400 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 msgr2=0x7fb52c1a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb51400e9f0 tx=0x7fb51400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 shutdown_connections 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb50c0a3ea0 0x7fb50c0a6360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb52c1150a0 0x7fb52c1a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb52c109870 0x7fb52c1a14c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 --2- 172.21.15.32:0/1456960261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52c108f50 0x7fb52c1a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 >> 172.21.15.32:0/1456960261 conn(0x7fb52c06e2e0 msgr2=0x7fb52c070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 shutdown_connections 2024-04-23T18:22:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.796+0000 7fb5311ad640 1 -- 172.21.15.32:0/1456960261 wait complete. 2024-04-23T18:22:08.805 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:22:08.805 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:22:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 400.0 s, 401.6 tps, lat 1564.017 ms stddev 6686.647 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 -- 172.21.15.32:0/726608468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70109870 msgr2=0x7f4f70114b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 --2- 172.21.15.32:0/726608468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70109870 0x7f4f70114b60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009860 tx=0x7f4f6002a3d0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 -- 172.21.15.32:0/726608468 shutdown_connections 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 --2- 172.21.15.32:0/726608468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f701150a0 0x7f4f70117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 --2- 172.21.15.32:0/726608468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70109870 0x7f4f70114b60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 --2- 172.21.15.32:0/726608468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f70108f50 0x7f4f70109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 -- 172.21.15.32:0/726608468 >> 172.21.15.32:0/726608468 conn(0x7f4f7006e540 msgr2=0x7f4f7006e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 -- 172.21.15.32:0/726608468 shutdown_connections 2024-04-23T18:22:08.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.980+0000 7f4f776fd640 1 -- 172.21.15.32:0/726608468 wait complete. 2024-04-23T18:22:08.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.981+0000 7f4f776fd640 1 Processor -- start 2024-04-23T18:22:08.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.981+0000 7f4f776fd640 1 -- start start 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f70109870 0x7f4f701a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f701150a0 0x7f4f701a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7011a920 con 0x7f4f701150a0 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7011a7a0 con 0x7f4f70108f50 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f776fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7011aaa0 con 0x7f4f70109870 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:39740/0 (socket says 172.21.15.32:39740) 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 -- 172.21.15.32:0/132218092 learned_addr learned my addr 172.21.15.32:0/132218092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f701150a0 0x7f4f701a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f74c98640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f70109870 0x7f4f701a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 -- 172.21.15.32:0/132218092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f70109870 msgr2=0x7f4f701a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f70109870 0x7f4f701a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 -- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f701150a0 msgr2=0x7f4f701a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f701150a0 0x7f4f701a79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.982+0000 7f4f75499640 1 -- 172.21.15.32:0/132218092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f701a80d0 con 0x7f4f70108f50 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f75499640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c420 tx=0x7f4f68009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f667fc640 1 -- 172.21.15.32:0/132218092 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f68003970 con 0x7f4f70108f50 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f701a83c0 con 0x7f4f70108f50 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f701afc50 con 0x7f4f70108f50 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f667fc640 1 -- 172.21.15.32:0/132218092 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f68004020 con 0x7f4f70108f50 2024-04-23T18:22:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.983+0000 7f4f667fc640 1 -- 172.21.15.32:0/132218092 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800b980 con 0x7f4f70108f50 2024-04-23T18:22:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.986+0000 7f4f667fc640 1 -- 172.21.15.32:0/132218092 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4f6801d020 con 0x7f4f70108f50 2024-04-23T18:22:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.987+0000 7f4f667fc640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 0x7f4f5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.987+0000 7f4f667fc640 1 -- 172.21.15.32:0/132218092 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f4f680d0540 con 0x7f4f70108f50 2024-04-23T18:22:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.987+0000 7f4f74c98640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 0x7f4f5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 Processor -- start 2024-04-23T18:22:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 -- start start 2024-04-23T18:22:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f74c98640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 0x7f4f5c0a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4f60031000 tx=0x7f4f600094f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9549b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a954c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5635a94a4370 con 0x5635a954a090 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5635a94412e0 con 0x5635a953f9c0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.988+0000 7f4f7a410740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5635a94a0330 con 0x5635a953f0a0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f74c98640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9549b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36178/0 (socket says 172.21.15.32:36178) 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 -- 172.21.15.32:0/1819456895 learned_addr learned my addr 172.21.15.32:0/1819456895 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a954c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 -- 172.21.15.32:0/1819456895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 msgr2=0x5635a9549b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9549b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 -- 172.21.15.32:0/1819456895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 msgr2=0x5635a954c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a954c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 -- 172.21.15.32:0/1819456895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635a9282d80 con 0x5635a953f0a0 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f75499640 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800ffd0 tx=0x7f4f6809cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42fdb313affa0973 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f5b7fe640 1 -- 172.21.15.32:0/1819456895 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680cdf10 con 0x5635a953f0a0 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f5b7fe640 1 -- 172.21.15.32:0/1819456895 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f68098b50 con 0x5635a953f0a0 2024-04-23T18:22:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.989+0000 7f4f5b7fe640 1 -- 172.21.15.32:0/1819456895 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680a4f00 con 0x5635a953f0a0 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.990+0000 7f4f7a410740 1 -- 172.21.15.32:0/1819456895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 msgr2=0x5635a953f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.990+0000 7f4f7a410740 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800ffd0 tx=0x7f4f6809cd20 comp rx=0 tx=0).stop 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.991+0000 7f4f7a410740 1 -- 172.21.15.32:0/1819456895 shutdown_connections 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.991+0000 7f4f7a410740 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a954c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.991+0000 7f4f7a410740 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9549b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.991+0000 7f4f7a410740 1 --2- 172.21.15.32:0/1819456895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a953f480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.991+0000 7f4f7a410740 1 -- 172.21.15.32:0/1819456895 >> 172.21.15.32:0/1819456895 conn(0x5635a953b3c0 msgr2=0x5635a953b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.992+0000 7f4f7a410740 1 -- 172.21.15.32:0/1819456895 shutdown_connections 2024-04-23T18:22:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.992+0000 7f4f7a410740 1 -- 172.21.15.32:0/1819456895 wait complete. 2024-04-23T18:22:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.993+0000 7f4f7a410740 1 -- start start 2024-04-23T18:22:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.993+0000 7f4f7a410740 1 Processor -- start 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.993+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a9546f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.993+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a954a090 0x5635a9547a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f7a410740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5635a954ce80 con 0x5635a953f0a0 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f7a410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5635a943efb0 con 0x5635a953f9c0 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f7a410740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5635a954d000 con 0x5635a954a090 2024-04-23T18:22:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f7a410740 10 client.0 _ll_register_callbacks cb 0x5635a943e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f75499640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a9546f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:39752/0 (socket says 172.21.15.32:39752) 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 -- 172.21.15.32:0/3165688905 learned_addr learned my addr 172.21.15.32:0/3165688905 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a954a090 0x5635a9547a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 -- 172.21.15.32:0/3165688905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a954a090 msgr2=0x5635a9547a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a954a090 0x5635a9547a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 -- 172.21.15.32:0/3165688905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 msgr2=0x5635a9546f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a9546f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 -- 172.21.15.32:0/3165688905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635a92e3510 con 0x5635a953f9c0 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f75499640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a9546f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f74c98640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009860 tx=0x7f4f6002a3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ec2166df5acdf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.994+0000 7f4f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:22:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.995+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f600046c0 con 0x5635a953f9c0 2024-04-23T18:22:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.995+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f4f60004860 con 0x5635a953f9c0 2024-04-23T18:22:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.996+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f68098b50 con 0x5635a953f9c0 2024-04-23T18:22:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.996+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4f60004a60 con 0x5635a953f9c0 2024-04-23T18:22:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.996+0000 7f4f7a410740 10 client.25964 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.996+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5635a9176670 con 0x5635a953f9c0 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.996+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004c00 con 0x5635a953f9c0 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f4f60044050 con 0x5635a953f9c0 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 10 client.25964 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 10 client.25964 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 20 client.25964 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a955c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 4 client.25964 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4f68098b50 con 0x5635a955a450 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f75499640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a955c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f58ff9640 10 client.25964 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.997+0000 7f4f75499640 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a955c880 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3be04d5c166d72f5 server_cookie=98fbaba18195a9a6 in_seq=0 out_seq=0 2024-04-23T18:22:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3165688905 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4f68098b50 con 0x5635a955a450 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f58ff9640 10 client.25964 handle_command_reply: tid=0 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 client.25964 shutdown 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 msgr2=0x5635a95474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009860 tx=0x7f4f6002a3d0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 shutdown_connections 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a955c880 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f58ff9640 0 client.25964 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a954a090 0x5635a9547a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a95474c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3165688905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a9546f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.998+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 >> 172.21.15.32:0/3165688905 conn(0x5635a953b3c0 msgr2=0x5635a953a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.999+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 shutdown_connections 2024-04-23T18:22:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.999+0000 7f4f7a410740 1 -- 172.21.15.32:0/3165688905 wait complete. 2024-04-23T18:22:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:08.999+0000 7f4f7a410740 20 client.25964 trim_cache size 0 max 16384 2024-04-23T18:22:09.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.024+0000 7f4f7a410740 1 Processor -- start 2024-04-23T18:22:09.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.024+0000 7f4f7a410740 1 -- start start 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.024+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a9555da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a9547650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f7a410740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f540110e0 con 0x5635a954a090 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f75499640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a9555da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f75499640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a9555da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:36196/0 (socket says 172.21.15.32:36196) 2024-04-23T18:22:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:39756/0 (socket says 172.21.15.32:39756) 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f75499640 1 -- 172.21.15.32:0/768091644 learned_addr learned my addr 172.21.15.32:0/768091644 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400e920 con 0x5635a953f9c0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400c770 con 0x5635a953f0a0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a9547650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 -- 172.21.15.32:0/768091644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 msgr2=0x5635a9555da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a9555da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 -- 172.21.15.32:0/768091644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 msgr2=0x5635a9547650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a9547650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 -- 172.21.15.32:0/768091644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635a9545f50 con 0x5635a953f9c0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.025+0000 7f4f74c98640 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009830 tx=0x7f4f600305c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6024bc0616d71699 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.026+0000 7f4f5affd640 1 -- 172.21.15.32:0/768091644 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60004800 con 0x5635a953f9c0 2024-04-23T18:22:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.026+0000 7f4f5affd640 1 -- 172.21.15.32:0/768091644 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004c00 con 0x5635a953f9c0 2024-04-23T18:22:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.026+0000 7f4f5affd640 1 -- 172.21.15.32:0/768091644 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60052710 con 0x5635a953f9c0 2024-04-23T18:22:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.026+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 msgr2=0x5635a9541800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.026+0000 7f4f7a410740 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009830 tx=0x7f4f600305c0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 shutdown_connections 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a954a090 0x5635a9547650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a953f9c0 0x5635a9541800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 --2- 172.21.15.32:0/768091644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f0a0 0x5635a9555da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 >> 172.21.15.32:0/768091644 conn(0x5635a953b3c0 msgr2=0x5635a9546f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 shutdown_connections 2024-04-23T18:22:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 -- 172.21.15.32:0/768091644 wait complete. 2024-04-23T18:22:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.028+0000 7f4f7a410740 1 -- start start 2024-04-23T18:22:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 Processor -- start 2024-04-23T18:22:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f9c0 0x5635a95527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 0x5635a954f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400cde0 con 0x5635a953f0a0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f75499640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400e250 con 0x5635a954a090 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400d4b0 con 0x5635a953f9c0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f75499640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35986/0 (socket says 172.21.15.32:35986) 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f75499640 1 -- 172.21.15.32:0/3926188997 learned_addr learned my addr 172.21.15.32:0/3926188997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 0x5635a954f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f7a410740 10 client.0 _ll_register_callbacks cb 0x7f4f60030df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.029+0000 7f4f74c98640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f9c0 0x5635a95527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 -- 172.21.15.32:0/3926188997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f9c0 msgr2=0x5635a95527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f9c0 0x5635a95527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 -- 172.21.15.32:0/3926188997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 msgr2=0x5635a95522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 -- 172.21.15.32:0/3926188997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635a954faf0 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75499640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f75c9a640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 0x5635a954f2c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5635a953c550 tx=0x7f4f6c012ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f62c1889bb996f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c023060 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8528+0+0 (secure 0 0 0) 0x7f4f6c004230 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.030+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f60004c00 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4f6c004430 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 10 client.25970 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5635a95c9dc0 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004c00 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f4f6c003ab0 con 0x5635a954a090 2024-04-23T18:22:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 10 client.25970 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 10 client.25970 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 20 client.25970 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a95cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 4 client.25970 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.031+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x5635a94b1d90 con 0x5635a955a450 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.032+0000 7f4f75499640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a95cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.033+0000 7f4f75499640 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a95cc650 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d9ce30fd78b32a7 server_cookie=cb32a1f825881a9b in_seq=0 out_seq=0 2024-04-23T18:22:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.033+0000 7f4f58ff9640 10 client.25970 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:09.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:08 smithi032 ceph-mon[29221]: pgmap v215: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 585 op/s 2024-04-23T18:22:09.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:08 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2610772494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:09.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:08 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1456960261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:09.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f58ff9640 1 -- 172.21.15.32:0/3926188997 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5635a94b1d90 con 0x5635a955a450 2024-04-23T18:22:09.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f58ff9640 10 client.25970 handle_command_reply: tid=0 2024-04-23T18:22:09.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 client.25970 shutdown 2024-04-23T18:22:09.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 msgr2=0x5635a954f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 0x5635a954f2c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5635a953c550 tx=0x7f4f6c012ca0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 shutdown_connections 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5635a955a450 0x5635a95cc650 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635a954a090 0x5635a954f2c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5635a953f9c0 0x5635a95527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 --2- 172.21.15.32:0/3926188997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5635a953f0a0 0x5635a95522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f58ff9640 0 client.25970 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 >> 172.21.15.32:0/3926188997 conn(0x5635a953b3c0 msgr2=0x5635a95477d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 shutdown_connections 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 1 -- 172.21.15.32:0/3926188997 wait complete. 2024-04-23T18:22:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.276+0000 7f4f7a410740 20 client.25970 trim_cache size 0 max 16384 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 37, 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 26, 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "e434819a": { 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 26, 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:22:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:09.274 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:22:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 msgr2=0x7f4f5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 0x7f4f5c0a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4f60031000 tx=0x7f4f600094f0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 msgr2=0x7f4f701a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c420 tx=0x7f4f68009f90 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 shutdown_connections 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4f5c0a3da0 0x7f4f5c0a6260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f701150a0 0x7f4f701a79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f70109870 0x7f4f701a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 --2- 172.21.15.32:0/132218092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f70108f50 0x7f4f701a30e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 >> 172.21.15.32:0/132218092 conn(0x7f4f7006e540 msgr2=0x7f4f7010a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 shutdown_connections 2024-04-23T18:22:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:09.277+0000 7f4f776fd640 1 -- 172.21.15.32:0/132218092 wait complete. 2024-04-23T18:22:09.281 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e434819a', quiesce took 1.1 seconds. Will release after: 30.0 2024-04-23T18:22:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:08 smithi040 ceph-mon[32626]: pgmap v215: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 585 op/s 2024-04-23T18:22:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:08 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2610772494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:08 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1456960261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:09.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:08 smithi064 ceph-mon[33062]: pgmap v215: 97 pgs: 97 active+clean; 4.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 585 op/s 2024-04-23T18:22:09.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:08 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2610772494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:09.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:08 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1456960261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:10 smithi032 ceph-mon[29221]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 1.13k op/s 2024-04-23T18:22:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:10 smithi032 ceph-mon[29221]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:22:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:10 smithi040 ceph-mon[32626]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 1.13k op/s 2024-04-23T18:22:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:10 smithi040 ceph-mon[32626]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:22:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:10 smithi064 ceph-mon[33062]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 1.13k op/s 2024-04-23T18:22:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:10 smithi064 ceph-mon[33062]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:22:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 -- 172.21.15.32:0/3738014767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f801010a0 msgr2=0x7f8f80101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 --2- 172.21.15.32:0/3738014767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f801010a0 0x7f8f80101480 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009ca0 tx=0x7f8f7002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 -- 172.21.15.32:0/3738014767 shutdown_connections 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 --2- 172.21.15.32:0/3738014767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f80106170 0x7f8f8010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 --2- 172.21.15.32:0/3738014767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80101a50 0x7f8f80105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 --2- 172.21.15.32:0/3738014767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f801010a0 0x7f8f80101480 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 -- 172.21.15.32:0/3738014767 >> 172.21.15.32:0/3738014767 conn(0x7f8f80077f70 msgr2=0x7f8f80078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 -- 172.21.15.32:0/3738014767 shutdown_connections 2024-04-23T18:22:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.190+0000 7f8f84c23640 1 -- 172.21.15.32:0/3738014767 wait complete. 2024-04-23T18:22:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 Processor -- start 2024-04-23T18:22:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 -- start start 2024-04-23T18:22:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f801010a0 0x7f8f801986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80106170 0x7f8f80198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801110b0 con 0x7f8f8019cfb0 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80110f30 con 0x7f8f80106170 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f84c23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80111230 con 0x7f8f801010a0 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f7ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f7ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36004/0 (socket says 172.21.15.32:36004) 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f7ed9d640 1 -- 172.21.15.32:0/2361351121 learned_addr learned my addr 172.21.15.32:0/2361351121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f801010a0 0x7f8f801986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.191+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80106170 0x7f8f80198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 -- 172.21.15.32:0/2361351121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f801010a0 msgr2=0x7f8f801986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f801010a0 0x7f8f801986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 -- 172.21.15.32:0/2361351121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80106170 msgr2=0x7f8f80198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80106170 0x7f8f80198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 -- 172.21.15.32:0/2361351121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8019daa0 con 0x7f8f8019cfb0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f7ed9d640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800e970 tx=0x7f8f7800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f6f7fe640 1 -- 172.21.15.32:0/2361351121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800ccc0 con 0x7f8f8019cfb0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8019dd90 con 0x7f8f8019cfb0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f6f7fe640 1 -- 172.21.15.32:0/2361351121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f7800ce60 con 0x7f8f8019cfb0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f6f7fe640 1 -- 172.21.15.32:0/2361351121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78010430 con 0x7f8f8019cfb0 2024-04-23T18:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.192+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f801a5530 con 0x7f8f8019cfb0 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.194+0000 7f8f6f7fe640 1 -- 172.21.15.32:0/2361351121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8f78004ff0 con 0x7f8f8019cfb0 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 Processor -- start 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 -- start start 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd37900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 0x558d0dd49000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558d0dc9c970 con 0x558d0dd37e40 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d0dc394a0 con 0x558d0dd46c10 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f87a15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558d0dc98930 con 0x558d0dd37520 2024-04-23T18:22:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f6f7fe640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 0x7f8f680a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36006/0 (socket says 172.21.15.32:36006) 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7e59c640 1 -- 172.21.15.32:0/1339448865 learned_addr learned my addr 172.21.15.32:0/1339448865 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7ed9d640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 0x7f8f680a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd37900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.195+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 0x558d0dd49000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f6f7fe640 1 -- 172.21.15.32:0/2361351121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f8f780d0c20 con 0x7f8f8019cfb0 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 -- 172.21.15.32:0/1339448865 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 msgr2=0x558d0dd37900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd37900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 -- 172.21.15.32:0/1339448865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 msgr2=0x558d0dd49000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 0x558d0dd49000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 -- 172.21.15.32:0/1339448865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0d9af210 con 0x558d0dd37e40 2024-04-23T18:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 0x558d0dd49000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7ed9d640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 0x7f8f680a61d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8f7809cb70 tx=0x7f8f78098850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8f70035000 tx=0x7f8f700093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de55e51239102366 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f6cff9640 1 -- 172.21.15.32:0/1339448865 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70031cf0 con 0x558d0dd37e40 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f6cff9640 1 -- 172.21.15.32:0/1339448865 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f70036030 con 0x558d0dd37e40 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.196+0000 7f8f6cff9640 1 -- 172.21.15.32:0/1339448865 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70004260 con 0x558d0dd37e40 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- 172.21.15.32:0/1339448865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 msgr2=0x558d0dd466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8f70035000 tx=0x7f8f700093f0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- 172.21.15.32:0/1339448865 shutdown_connections 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd46c10 0x558d0dd49000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37e40 0x558d0dd466d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 --2- 172.21.15.32:0/1339448865 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd37900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- 172.21.15.32:0/1339448865 >> 172.21.15.32:0/1339448865 conn(0x558d0dd33b30 msgr2=0x558d0dd33f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- 172.21.15.32:0/1339448865 shutdown_connections 2024-04-23T18:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- 172.21.15.32:0/1339448865 wait complete. 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 -- start start 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.197+0000 7f8f87a15740 1 Processor -- start 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd43800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd46c10 0x558d0dd43d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558d0dd49660 con 0x558d0dd37520 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d0dc370b0 con 0x558d0dd37e40 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558d0dd497e0 con 0x558d0dd46c10 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 10 client.0 _ll_register_callbacks cb 0x558d0dc24d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd43800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36016/0 (socket says 172.21.15.32:36016) 2024-04-23T18:22:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2008499801 learned_addr learned my addr 172.21.15.32:0/2008499801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2008499801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd46c10 msgr2=0x558d0dd43d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd46c10 0x558d0dd43d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 msgr2=0x558d0dd43800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd43800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2008499801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0da7b620 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400ce40 tx=0x7f8f74011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65bbb85b779bcd8c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74011e80 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f8f74004030 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.198+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f70036030 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8f74024050 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 10 client.16392 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558d0dd555b0 con 0x558d0dd37520 2024-04-23T18:22:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f70036030 con 0x558d0dd37520 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f8f74003ce0 con 0x558d0dd37520 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 10 client.16392 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 10 client.16392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 20 client.16392 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0dd56a70 0x558d0dd58e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 4 client.16392 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d0dcaa390 con 0x558d0dd56a70 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.199+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0dd56a70 0x558d0dd58e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.200+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0dd56a70 0x558d0dd58e60 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96caf80373c4809 server_cookie=4825d9b9b95c2cb9 in_seq=0 out_seq=0 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.200+0000 7f8f65ffb640 10 client.16392 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.200+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2008499801 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d0dcaa390 con 0x558d0dd56a70 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f65ffb640 10 client.16392 handle_command_reply: tid=0 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 client.16392 shutdown 2024-04-23T18:22:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 msgr2=0x558d0dd46510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400ce40 tx=0x7f8f74011810 comp rx=0 tx=0).stop 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 shutdown_connections 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0dd56a70 0x558d0dd58e60 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd46c10 0x558d0dd43d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd43800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f65ffb640 0 client.16392 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2008499801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd37520 0x558d0dd46510 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 >> 172.21.15.32:0/2008499801 conn(0x558d0dd33b30 msgr2=0x558d0dd399a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 shutdown_connections 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 1 -- 172.21.15.32:0/2008499801 wait complete. 2024-04-23T18:22:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.201+0000 7f8f87a15740 20 client.16392 trim_cache size 0 max 16384 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.216+0000 7f8f87a15740 1 Processor -- start 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.216+0000 7f8f87a15740 1 -- start start 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37520 0x558d0dd52520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37e40 0x558d0dd3ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ae1c0 con 0x558d0dd46c10 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801aba00 con 0x558d0dd37520 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f87a15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9890 con 0x558d0dd37e40 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36026/0 (socket says 172.21.15.32:36026) 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/4191043215 learned_addr learned my addr 172.21.15.32:0/4191043215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/4191043215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37e40 msgr2=0x558d0dd3ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37e40 0x558d0dd3ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/4191043215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37520 msgr2=0x558d0dd52520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37520 0x558d0dd52520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/4191043215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0dd33960 con 0x558d0dd46c10 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8f74012690 tx=0x7f8f74005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462298717a0373d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f67fff640 1 -- 172.21.15.32:0/4191043215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7402d020 con 0x558d0dd46c10 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f67fff640 1 -- 172.21.15.32:0/4191043215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f740054e0 con 0x558d0dd46c10 2024-04-23T18:22:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.217+0000 7f8f67fff640 1 -- 172.21.15.32:0/4191043215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74004c30 con 0x558d0dd46c10 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- 172.21.15.32:0/4191043215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 msgr2=0x558d0dd46040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8f74012690 tx=0x7f8f74005c50 comp rx=0 tx=0).stop 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- 172.21.15.32:0/4191043215 shutdown_connections 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0dd46040 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37e40 0x558d0dd3ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 --2- 172.21.15.32:0/4191043215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37520 0x558d0dd52520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- 172.21.15.32:0/4191043215 >> 172.21.15.32:0/4191043215 conn(0x558d0dd33b30 msgr2=0x558d0dd3e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- 172.21.15.32:0/4191043215 shutdown_connections 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- 172.21.15.32:0/4191043215 wait complete. 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.218+0000 7f8f87a15740 1 -- start start 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 Processor -- start 2024-04-23T18:22:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd4a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9ee0 con 0x558d0dd46c10 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ab330 con 0x558d0dd37e40 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801aa590 con 0x558d0dd37520 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f87a15740 10 client.0 _ll_register_callbacks cb 0x7f8f740148a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36038/0 (socket says 172.21.15.32:36038) 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2723928546 learned_addr learned my addr 172.21.15.32:0/2723928546 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd4a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2723928546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 msgr2=0x558d0dd49d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd49d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 msgr2=0x558d0dd4a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd4a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 -- 172.21.15.32:0/2723928546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0ddb56a0 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f7dd9b640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8f74011d90 tx=0x7f8f74016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e9c8108cf69a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.219+0000 7f8f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7403a060 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f8f74004030 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f740054e0 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8f7402d030 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 10 client.16398 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558d0ddbca20 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f740054e0 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f8f74032070 con 0x558d0dd46c10 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 10 client.16398 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 10 client.16398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 20 client.16398 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0ddbd060 0x558d0ddbf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 4 client.16398 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.220+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d0dd440d0 con 0x558d0ddbd060 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.221+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0ddbd060 0x558d0ddbf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.221+0000 7f8f7e59c640 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0ddbd060 0x558d0ddbf480 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e14a8eee46bd4a7 server_cookie=1ebf4c6cc769880 in_seq=0 out_seq=0 2024-04-23T18:22:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.221+0000 7f8f65ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f65ffb640 1 -- 172.21.15.32:0/2723928546 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d0dd440d0 con 0x558d0ddbd060 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f65ffb640 10 client.16398 handle_command_reply: tid=0 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 client.16398 shutdown 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 msgr2=0x558d0ddb5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8f74011d90 tx=0x7f8f74016040 comp rx=0 tx=0).stop 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 shutdown_connections 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x558d0ddbd060 0x558d0ddbf480 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558d0dd46c10 0x558d0ddb5060 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f65ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d0dd37e40 0x558d0dd4a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 --2- 172.21.15.32:0/2723928546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558d0dd37520 0x558d0dd49d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 >> 172.21.15.32:0/2723928546 conn(0x558d0dd33b30 msgr2=0x558d0dd39950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 shutdown_connections 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 1 -- 172.21.15.32:0/2723928546 wait complete. 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.234+0000 7f8f87a15740 20 client.16398 trim_cache size 0 max 16384 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:22:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 msgr2=0x7f8f680a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 0x7f8f680a61d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8f7809cb70 tx=0x7f8f78098850 comp rx=0 tx=0).stop 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 msgr2=0x7f8f8019d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800e970 tx=0x7f8f7800ed70 comp rx=0 tx=0).stop 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 shutdown_connections 2024-04-23T18:22:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8f680a3d10 0x7f8f680a61d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f8019cfb0 0x7f8f8019d460 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f80106170 0x7f8f80198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 --2- 172.21.15.32:0/2361351121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f801010a0 0x7f8f801986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 >> 172.21.15.32:0/2361351121 conn(0x7f8f80077f70 msgr2=0x7f8f80069ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 shutdown_connections 2024-04-23T18:22:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:13.235+0000 7f8f84c23640 1 -- 172.21.15.32:0/2361351121 wait complete. 2024-04-23T18:22:13.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:12 smithi032 ceph-mon[29221]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 MiB/s wr, 938 op/s 2024-04-23T18:22:13.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:12 smithi040 ceph-mon[32626]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 MiB/s wr, 938 op/s 2024-04-23T18:22:13.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:12 smithi064 ceph-mon[33062]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 MiB/s wr, 938 op/s 2024-04-23T18:22:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 405.0 s, 84.2 tps, lat 44.452 ms stddev 38.939 2024-04-23T18:22:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:14 smithi032 ceph-mon[29221]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 735 op/s 2024-04-23T18:22:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:14 smithi040 ceph-mon[32626]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 735 op/s 2024-04-23T18:22:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:14 smithi064 ceph-mon[33062]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 735 op/s 2024-04-23T18:22:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:16 smithi032 ceph-mon[29221]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 MiB/s wr, 644 op/s 2024-04-23T18:22:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:16 smithi040 ceph-mon[32626]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 MiB/s wr, 644 op/s 2024-04-23T18:22:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:16 smithi064 ceph-mon[33062]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 MiB/s wr, 644 op/s 2024-04-23T18:22:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 410.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:19.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:19 smithi032 ceph-mon[29221]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 16 MiB/s wr, 613 op/s 2024-04-23T18:22:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:19 smithi040 ceph-mon[32626]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 16 MiB/s wr, 613 op/s 2024-04-23T18:22:19.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:19 smithi064 ceph-mon[33062]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 16 MiB/s wr, 613 op/s 2024-04-23T18:22:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:21 smithi032 ceph-mon[29221]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 633 KiB/s wr, 64 op/s 2024-04-23T18:22:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:21 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:21 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:21 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:21 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:22:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:21 smithi040 ceph-mon[32626]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 633 KiB/s wr, 64 op/s 2024-04-23T18:22:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:21 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:21 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:21 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:21 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:22:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:21 smithi064 ceph-mon[33062]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 633 KiB/s wr, 64 op/s 2024-04-23T18:22:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:21 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:21 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:21 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:21 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:22:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:23 smithi032 ceph-mon[29221]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 582 KiB/s wr, 59 op/s 2024-04-23T18:22:23.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:23 smithi040 ceph-mon[32626]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 582 KiB/s wr, 59 op/s 2024-04-23T18:22:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:23 smithi064 ceph-mon[33062]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 582 KiB/s wr, 59 op/s 2024-04-23T18:22:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 415.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:25.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:25 smithi032 ceph-mon[29221]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-23T18:22:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:25 smithi040 ceph-mon[32626]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-23T18:22:25.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:25 smithi064 ceph-mon[33062]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- 172.21.15.32:0/1932422479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 msgr2=0x7ff4841035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 --2- 172.21.15.32:0/1932422479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff4841035a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff474009900 tx=0x7ff47402a480 comp rx=0 tx=0).stop 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- 172.21.15.32:0/1932422479 shutdown_connections 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 --2- 172.21.15.32:0/1932422479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff48410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 --2- 172.21.15.32:0/1932422479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 0x7ff48410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 --2- 172.21.15.32:0/1932422479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff4841035a0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- 172.21.15.32:0/1932422479 >> 172.21.15.32:0/1932422479 conn(0x7ff484077f70 msgr2=0x7ff484078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- 172.21.15.32:0/1932422479 shutdown_connections 2024-04-23T18:22:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- 172.21.15.32:0/1932422479 wait complete. 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 Processor -- start 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.335+0000 7ff4893ad640 1 -- start start 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 0x7ff484101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff484101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841121e0 con 0x7ff4841031c0 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff484112060 con 0x7ff48410c590 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4893ad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff484112360 con 0x7ff484103ae0 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35244/0 (socket says 172.21.15.32:35244) 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 -- 172.21.15.32:0/2900196395 learned_addr learned my addr 172.21.15.32:0/2900196395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff47bfff640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 0x7ff484101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4837fe640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff484101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 -- 172.21.15.32:0/2900196395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 msgr2=0x7ff484101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 0x7ff484101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 -- 172.21.15.32:0/2900196395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 msgr2=0x7ff484101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff484101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 -- 172.21.15.32:0/2900196395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4841a6c80 con 0x7ff4841031c0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff4837fe640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff484101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.336+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff474005940 tx=0x7ff4740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.337+0000 7ff480ff9640 1 -- 172.21.15.32:0/2900196395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47403f030 con 0x7ff4841031c0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.337+0000 7ff480ff9640 1 -- 172.21.15.32:0/2900196395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff474003cb0 con 0x7ff4841031c0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.337+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4841a6f10 con 0x7ff4841031c0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.337+0000 7ff480ff9640 1 -- 172.21.15.32:0/2900196395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474035070 con 0x7ff4841031c0 2024-04-23T18:22:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.337+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4841a7370 con 0x7ff4841031c0 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.339+0000 7ff480ff9640 1 -- 172.21.15.32:0/2900196395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff474042030 con 0x7ff4841031c0 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.339+0000 7ff48c1a3740 1 Processor -- start 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.339+0000 7ff48c1a3740 1 -- start start 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 0x556bbe8aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8b6800 0x556bbe8b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556bbe810970 con 0x556bbe8abf40 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bbe7ad4a0 con 0x556bbe8b6800 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556bbe80c930 con 0x556bbe8ab620 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff482ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff482ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35252/0 (socket says 172.21.15.32:35252) 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff482ffd640 1 -- 172.21.15.32:0/1682392370 learned_addr learned my addr 172.21.15.32:0/1682392370 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 0x556bbe8aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff480ff9640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 0x7ff4700a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff47bfff640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8b6800 0x556bbe8b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.340+0000 7ff4837fe640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 0x7ff4700a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 -- 172.21.15.32:0/1682392370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8b6800 msgr2=0x556bbe8b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8b6800 0x556bbe8b8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff480ff9640 1 -- 172.21.15.32:0/2900196395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7ff4740f1090 con 0x7ff4841031c0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 -- 172.21.15.32:0/1682392370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 msgr2=0x556bbe8b62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 -- 172.21.15.32:0/1682392370 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bbe523210 con 0x556bbe8ab620 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 0x556bbe8aba00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff47c012860 tx=0x7ff47c012d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dcd3bb5a2e2cb6d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4797fa640 1 -- 172.21.15.32:0/1682392370 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c022070 con 0x556bbe8ab620 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4837fe640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 0x7ff4700a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff47c020000 tx=0x7ff47c01f000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4797fa640 1 -- 172.21.15.32:0/1682392370 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c01eae0 con 0x556bbe8ab620 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.341+0000 7ff4797fa640 1 -- 172.21.15.32:0/1682392370 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c005520 con 0x556bbe8ab620 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1682392370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 msgr2=0x556bbe8aba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 0x556bbe8aba00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff47c012860 tx=0x7ff47c012d20 comp rx=0 tx=0).stop 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1682392370 shutdown_connections 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8b6800 0x556bbe8b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1682392370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8ab620 0x556bbe8aba00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1682392370 >> 172.21.15.32:0/1682392370 conn(0x556bbe8a7b30 msgr2=0x556bbe8a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1682392370 shutdown_connections 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1682392370 wait complete. 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 -- start start 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.342+0000 7ff48c1a3740 1 Processor -- start 2024-04-23T18:22:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8b6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe8b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556bbe8b9430 con 0x556bbe8abf40 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bbe7ab0b0 con 0x556bbe8ab620 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556bbe8b95b0 con 0x556bbe8b6800 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 10 client.0 _ll_register_callbacks cb 0x556bbe798d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35260/0 (socket says 172.21.15.32:35260) 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 -- 172.21.15.32:0/1128501782 learned_addr learned my addr 172.21.15.32:0/1128501782 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe8b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 -- 172.21.15.32:0/1128501782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 msgr2=0x556bbe8b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe8b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 -- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 msgr2=0x556bbe8b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 -- 172.21.15.32:0/1128501782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bbe5ef620 con 0x556bbe8abf40 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff47bfff640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b990 tx=0x7ff46c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9361a5d992538e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff46a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c005020 con 0x556bbe8abf40 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7ff46c0051c0 con 0x556bbe8abf40 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.343+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff47c01eae0 con 0x556bbe8abf40 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff46c0053c0 con 0x556bbe8abf40 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 10 client.16410 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556bbe8c5430 con 0x556bbe8abf40 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c01eae0 con 0x556bbe8abf40 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7ff46c022020 con 0x556bbe8abf40 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 10 client.16410 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 20 client.16410 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe8c6910 0x556bbe8c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 4 client.16410 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.344+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bbe81e390 con 0x556bbe8c6910 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.345+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe8c6910 0x556bbe8c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.345+0000 7ff46a7fc640 10 client.16410 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.345+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe8c6910 0x556bbe8c8d20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49f8431d6c5999 server_cookie=452cea31fad49602 in_seq=0 out_seq=0 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff46a7fc640 1 -- 172.21.15.32:0/1128501782 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bbe81e390 con 0x556bbe8c6910 2024-04-23T18:22:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff46a7fc640 10 client.16410 handle_command_reply: tid=0 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 client.16410 shutdown 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 msgr2=0x556bbe8b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b990 tx=0x7ff46c00be50 comp rx=0 tx=0).stop 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 shutdown_connections 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe8c6910 0x556bbe8c8d20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe8b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8b3440 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff46a7fc640 0 client.16410 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1128501782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 >> 172.21.15.32:0/1128501782 conn(0x556bbe8a7b30 msgr2=0x556bbe8ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 shutdown_connections 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1128501782 wait complete. 2024-04-23T18:22:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.346+0000 7ff48c1a3740 20 client.16410 trim_cache size 0 max 16384 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 Processor -- start 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 -- start start 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8abf40 0x556bbe8a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841b2670 con 0x556bbe8b6800 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841afeb0 con 0x556bbe8ab620 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.361+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841add20 con 0x556bbe8abf40 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35274/0 (socket says 172.21.15.32:35274) 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 -- 172.21.15.32:0/1638336666 learned_addr learned my addr 172.21.15.32:0/1638336666 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff47bfff640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8abf40 0x556bbe8a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 -- 172.21.15.32:0/1638336666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8abf40 msgr2=0x556bbe8a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8abf40 0x556bbe8a6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 -- 172.21.15.32:0/1638336666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 msgr2=0x556bbe8aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 -- 172.21.15.32:0/1638336666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bbe8abd70 con 0x556bbe8b6800 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff482ffd640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff4837fe640 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff47c021040 tx=0x7ff47c012710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680b42c0566b3e77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff478ff9640 1 -- 172.21.15.32:0/1638336666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c022040 con 0x556bbe8b6800 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff478ff9640 1 -- 172.21.15.32:0/1638336666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c012460 con 0x556bbe8b6800 2024-04-23T18:22:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.362+0000 7ff478ff9640 1 -- 172.21.15.32:0/1638336666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c026690 con 0x556bbe8b6800 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1638336666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 msgr2=0x556bbe8b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff47c021040 tx=0x7ff47c012710 comp rx=0 tx=0).stop 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1638336666 shutdown_connections 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8b6800 0x556bbe8b5c80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8abf40 0x556bbe8a6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/1638336666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1638336666 >> 172.21.15.32:0/1638336666 conn(0x556bbe8a7b30 msgr2=0x556bbe8a6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1638336666 shutdown_connections 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- 172.21.15.32:0/1638336666 wait complete. 2024-04-23T18:22:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 -- start start 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.363+0000 7ff48c1a3740 1 Processor -- start 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8ba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe92bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841ae370 con 0x556bbe8abf40 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841af7e0 con 0x556bbe8ab620 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4841aea40 con 0x556bbe8b6800 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff48c1a3740 10 client.0 _ll_register_callbacks cb 0x7ff46c0045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35288/0 (socket says 172.21.15.32:35288) 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 -- 172.21.15.32:0/2383815458 learned_addr learned my addr 172.21.15.32:0/2383815458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8ba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff4837fe640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe92bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 -- 172.21.15.32:0/2383815458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 msgr2=0x556bbe92bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe92bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 -- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 msgr2=0x556bbe8ba5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8ba5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 -- 172.21.15.32:0/2383815458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bbe8bb3b0 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff47bfff640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0044f0 tx=0x7ff46c01b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1f3e3ca9e4b546 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.364+0000 7ff46a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c005560 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7ff46c024460 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff47c012460 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff46c024660 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 10 client.16416 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556bbe92d4b0 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c012460 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7ff46c034070 con 0x556bbe8abf40 2024-04-23T18:22:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 10 client.16416 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 20 client.16416 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe92dc30 0x556bbe930050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 4 client.16416 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556bbe818200 con 0x556bbe92dc30 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.365+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe92dc30 0x556bbe930050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.366+0000 7ff482ffd640 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe92dc30 0x556bbe930050 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454138cbe4ebdaf5 server_cookie=afde6956a6347817 in_seq=0 out_seq=0 2024-04-23T18:22:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.366+0000 7ff46a7fc640 10 client.16416 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff46a7fc640 1 -- 172.21.15.32:0/2383815458 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556bbe818200 con 0x556bbe92dc30 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff46a7fc640 10 client.16416 handle_command_reply: tid=0 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 client.16416 shutdown 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 msgr2=0x556bbe8baae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0044f0 tx=0x7ff46c01b470 comp rx=0 tx=0).stop 2024-04-23T18:22:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 shutdown_connections 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556bbe92dc30 0x556bbe930050 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556bbe8b6800 0x556bbe92bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff46a7fc640 0 client.16416 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556bbe8abf40 0x556bbe8baae0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 --2- 172.21.15.32:0/2383815458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bbe8ab620 0x556bbe8ba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.377+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 >> 172.21.15.32:0/2383815458 conn(0x556bbe8a7b30 msgr2=0x556bbe8c1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 shutdown_connections 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff48c1a3740 1 -- 172.21.15.32:0/2383815458 wait complete. 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff48c1a3740 20 client.16416 trim_cache size 0 max 16384 2024-04-23T18:22:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:22:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:22:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:22:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:22:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 msgr2=0x7ff4700a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 0x7ff4700a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff47c020000 tx=0x7ff47c01f000 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 msgr2=0x7ff484100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff474005940 tx=0x7ff4740042f0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 shutdown_connections 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff48410c590 0x7ff484101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484103ae0 0x7ff484101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff4700a3d10 0x7ff4700a61d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 --2- 172.21.15.32:0/2900196395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4841031c0 0x7ff484100e10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 >> 172.21.15.32:0/2900196395 conn(0x7ff484077f70 msgr2=0x7ff4840fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 shutdown_connections 2024-04-23T18:22:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:26.378+0000 7ff4893ad640 1 -- 172.21.15.32:0/2900196395 wait complete. 2024-04-23T18:22:27.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:27 smithi032 ceph-mon[29221]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-04-23T18:22:27.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:27 smithi040 ceph-mon[32626]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-04-23T18:22:27.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:27 smithi064 ceph-mon[33062]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-04-23T18:22:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 420.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:29.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:29 smithi040 ceph-mon[32626]: pgmap v226: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-23T18:22:29.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:29 smithi064 ceph-mon[33062]: pgmap v226: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-23T18:22:29.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:29 smithi032 ceph-mon[29221]: pgmap v226: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-23T18:22:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:31 smithi040 ceph-mon[32626]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 261 op/s 2024-04-23T18:22:31.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:31 smithi064 ceph-mon[33062]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 261 op/s 2024-04-23T18:22:31.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:31 smithi032 ceph-mon[29221]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 261 op/s 2024-04-23T18:22:33.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:33 smithi040 ceph-mon[32626]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-23T18:22:33.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:33 smithi064 ceph-mon[33062]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-23T18:22:33.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:33 smithi032 ceph-mon[29221]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-23T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 425.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:35.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:35 smithi040 ceph-mon[32626]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.8 MiB/s wr, 260 op/s 2024-04-23T18:22:35.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:35 smithi064 ceph-mon[33062]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.8 MiB/s wr, 260 op/s 2024-04-23T18:22:35.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:35 smithi032 ceph-mon[29221]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.8 MiB/s wr, 260 op/s 2024-04-23T18:22:37.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:37 smithi040 ceph-mon[32626]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-04-23T18:22:37.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:37 smithi064 ceph-mon[33062]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-04-23T18:22:37.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:37 smithi032 ceph-mon[29221]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-04-23T18:22:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 430.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:22:39.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:39.309 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:39 smithi032 ceph-mon[29221]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-23T18:22:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:39 smithi040 ceph-mon[32626]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 -- 172.21.15.32:0/2615798858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8069250 msgr2=0x7f67b80696d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 --2- 172.21.15.32:0/2615798858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8069250 0x7f67b80696d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009900 tx=0x7f67a402a410 comp rx=0 tx=0).stop 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 -- 172.21.15.32:0/2615798858 shutdown_connections 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 --2- 172.21.15.32:0/2615798858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069c10 0x7f67b810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 --2- 172.21.15.32:0/2615798858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8069250 0x7f67b80696d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 --2- 172.21.15.32:0/2615798858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8106f50 0x7f67b8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 -- 172.21.15.32:0/2615798858 >> 172.21.15.32:0/2615798858 conn(0x7f67b8077c40 msgr2=0x7f67b8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 -- 172.21.15.32:0/2615798858 shutdown_connections 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.405+0000 7f67bf01e640 1 -- 172.21.15.32:0/2615798858 wait complete. 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 Processor -- start 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 -- start start 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069250 0x7f67b8101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8069c10 0x7f67b81016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67b81121f0 con 0x7f67b8106f50 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8112070 con 0x7f67b8069250 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bf01e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8112370 con 0x7f67b8069c10 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bd5bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bd5bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45832/0 (socket says 172.21.15.32:45832) 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bd5bb640 1 -- 172.21.15.32:0/779231699 learned_addr learned my addr 172.21.15.32:0/779231699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67b7fff640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8069c10 0x7f67b81016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bcdba640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069250 0x7f67b8101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bd5bb640 1 -- 172.21.15.32:0/779231699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8069c10 msgr2=0x7f67b81016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.406+0000 7f67bd5bb640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8069c10 0x7f67b81016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bd5bb640 1 -- 172.21.15.32:0/779231699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069250 msgr2=0x7f67b8101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bd5bb640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069250 0x7f67b8101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bd5bb640 1 -- 172.21.15.32:0/779231699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b81a6bf0 con 0x7f67b8106f50 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bd5bb640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f67b000cad0 tx=0x7f67b000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b000ae20 con 0x7f67b8106f50 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b81a6ee0 con 0x7f67b8106f50 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67b0005020 con 0x7f67b8106f50 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b000f6b0 con 0x7f67b8106f50 2024-04-23T18:22:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.407+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b81a73b0 con 0x7f67b8106f50 2024-04-23T18:22:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.408+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b8068e00 con 0x7f67b8106f50 2024-04-23T18:22:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.410+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f67b00051d0 con 0x7f67b8106f50 2024-04-23T18:22:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.410+0000 7f67b5ffb640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 0x7f67a80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.410+0000 7f67bcdba640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 0x7f67a80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.410+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f67b00d0ae0 con 0x7f67b8106f50 2024-04-23T18:22:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.411+0000 7f67bcdba640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 0x7f67a80a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005fd0 tx=0x7f67ac00b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.419+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67b0014030 con 0x7f67b8106f50 2024-04-23T18:22:39.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:39 smithi064 ceph-mon[33062]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-23T18:22:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/2205777840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d8101a50 msgr2=0x7fb7d8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/2205777840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d8101a50 0x7fb7d8105a90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009900 tx=0x7fb7c802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/2205777840 shutdown_connections 2024-04-23T18:22:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/2205777840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d8106170 0x7fb7d810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/2205777840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d8101a50 0x7fb7d8105a90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/2205777840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d81010a0 0x7fb7d8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/2205777840 >> 172.21.15.32:0/2205777840 conn(0x7fb7d8077f70 msgr2=0x7fb7d8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/2205777840 shutdown_connections 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.476+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/2205777840 wait complete. 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 Processor -- start 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 -- start start 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 0x7fb7d8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d819bed0 0x7fb7d819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81110b0 con 0x7fb7d81010a0 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8110f30 con 0x7fb7d8106170 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dfdaf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8111230 con 0x7fb7d819bed0 2024-04-23T18:22:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7ddb4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7ddb4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45854/0 (socket says 172.21.15.32:45854) 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7ddb4b640 1 -- 172.21.15.32:0/4139943095 learned_addr learned my addr 172.21.15.32:0/4139943095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7de34c640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d819bed0 0x7fb7d819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.477+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 0x7fb7d8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 -- 172.21.15.32:0/4139943095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d819bed0 msgr2=0x7fb7d819c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d819bed0 0x7fb7d819c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 -- 172.21.15.32:0/4139943095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 msgr2=0x7fb7d8197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 0x7fb7d8197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 -- 172.21.15.32:0/4139943095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d819c9c0 con 0x7fb7d81010a0 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 0x7fb7d8197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009970 tx=0x7fb7d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ceffd640 1 -- 172.21.15.32:0/4139943095 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400d020 con 0x7fb7d81010a0 2024-04-23T18:22:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ceffd640 1 -- 172.21.15.32:0/4139943095 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d400d1c0 con 0x7fb7d81010a0 2024-04-23T18:22:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d819ccb0 con 0x7fb7d81010a0 2024-04-23T18:22:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7ceffd640 1 -- 172.21.15.32:0/4139943095 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400cc50 con 0x7fb7d81010a0 2024-04-23T18:22:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.478+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d8069a90 con 0x7fb7d81010a0 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.480+0000 7fb7ceffd640 1 -- 172.21.15.32:0/4139943095 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb7d4007860 con 0x7fb7d81010a0 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 Processor -- start 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 -- start start 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad64d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad5b2970 con 0x55b4ad64de20 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad54f4a0 con 0x55b4ad65cc10 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad5ae930 con 0x55b4ad64d500 2024-04-23T18:22:39.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7ddb4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7ddb4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45856/0 (socket says 172.21.15.32:45856) 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7ddb4b640 1 -- 172.21.15.32:0/2239640698 learned_addr learned my addr 172.21.15.32:0/2239640698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7ceffd640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 0x7fb7c40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.481+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7de34c640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad64d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7de34c640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 0x7fb7c40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 -- 172.21.15.32:0/2239640698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 msgr2=0x55b4ad64d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7ceffd640 1 -- 172.21.15.32:0/4139943095 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fb7d40d1c30 con 0x7fb7d81010a0 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad64d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 -- 172.21.15.32:0/2239640698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 msgr2=0x55b4ad65c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 -- 172.21.15.32:0/2239640698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4ad2c5210 con 0x55b4ad65cc10 2024-04-23T18:22:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7de34c640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad64d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65f000 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8100a00 tx=0x7fb7c8032c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b074c0fc4083c0ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7de34c640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 0x7fb7c40a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0005fd0 tx=0x7fb7d0009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7c3fff640 1 -- 172.21.15.32:0/2239640698 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035030 con 0x55b4ad65cc10 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7c3fff640 1 -- 172.21.15.32:0/2239640698 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8032e20 con 0x55b4ad65cc10 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.482+0000 7fb7c3fff640 1 -- 172.21.15.32:0/2239640698 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80054b0 con 0x55b4ad65cc10 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/2239640698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 msgr2=0x55b4ad65f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65f000 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8100a00 tx=0x7fb7c8032c30 comp rx=0 tx=0).stop 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/2239640698 shutdown_connections 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65f000 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad65c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/2239640698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad64d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/2239640698 >> 172.21.15.32:0/2239640698 conn(0x55b4ad649b30 msgr2=0x55b4ad649f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/2239640698 shutdown_connections 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.483+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/2239640698 wait complete. 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 -- start start 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 Processor -- start 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad64d500 0x55b4ad65c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad65cc10 0x55b4ad659d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad65f7e0 con 0x55b4ad64de20 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad54d0b0 con 0x55b4ad64d500 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b4ad65f960 con 0x55b4ad65cc10 2024-04-23T18:22:39.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7e2b9c740 10 client.0 _ll_register_callbacks cb 0x55b4ad53ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45860/0 (socket says 172.21.15.32:45860) 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3559123198 learned_addr learned my addr 172.21.15.32:0/3559123198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad64d500 0x55b4ad65c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.484+0000 7fb7de34c640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad65cc10 0x55b4ad659d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3559123198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad65cc10 msgr2=0x55b4ad659d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad65cc10 0x55b4ad659d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad64d500 msgr2=0x55b4ad65c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad64d500 0x55b4ad65c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3559123198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4ad391620 con 0x55b4ad64de20 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8006010 tx=0x7fb7c80316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23732afe2e4b927 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035070 con 0x55b4ad64de20 2024-04-23T18:22:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fb7c8005440 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7c8032e20 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb7c8005640 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7e2b9c740 10 client.16434 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b4ad66b6f0 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8032e20 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.485+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fb7c8054070 con 0x55b4ad64de20 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 10 client.16434 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 10 client.16434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 20 client.16434 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad66ca40 0x55b4ad66ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 4 client.16434 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4ad5c0390 con 0x55b4ad66ca40 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad66ca40 0x55b4ad66ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad66ca40 0x55b4ad66ee30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffe1e3b60ed60edd server_cookie=6e4c8329093fed2 in_seq=0 out_seq=0 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.486+0000 7fb7c17fa640 10 client.16434 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3559123198 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4ad5c0390 con 0x55b4ad66ca40 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7c17fa640 10 client.16434 handle_command_reply: tid=0 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 client.16434 shutdown 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 msgr2=0x55b4ad6597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8006010 tx=0x7fb7c80316d0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 shutdown_connections 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad66ca40 0x55b4ad66ee30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad65cc10 0x55b4ad659d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad6597c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3559123198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad64d500 0x55b4ad65c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7c17fa640 0 client.16434 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 >> 172.21.15.32:0/3559123198 conn(0x55b4ad649b30 msgr2=0x55b4ad64f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 shutdown_connections 2024-04-23T18:22:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3559123198 wait complete. 2024-04-23T18:22:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.487+0000 7fb7e2b9c740 20 client.16434 trim_cache size 0 max 16384 2024-04-23T18:22:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 Processor -- start 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 -- start start 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad6686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81ae2b0 con 0x55b4ad64de20 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81abaf0 con 0x55b4ad65cc10 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.502+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81a9960 con 0x55b4ad64d500 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45864/0 (socket says 172.21.15.32:45864) 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 -- 172.21.15.32:0/1528502544 learned_addr learned my addr 172.21.15.32:0/1528502544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7de34c640 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad6686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 -- 172.21.15.32:0/1528502544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 msgr2=0x55b4ad6686e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad6686e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 -- 172.21.15.32:0/1528502544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 msgr2=0x55b4ad65c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 -- 172.21.15.32:0/1528502544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4ad649960 con 0x55b4ad64de20 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8004f00 tx=0x7fb7c8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7d879678ada111 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7c37fe640 1 -- 172.21.15.32:0/1528502544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8003670 con 0x55b4ad64de20 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7c37fe640 1 -- 172.21.15.32:0/1528502544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8005440 con 0x55b4ad64de20 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.503+0000 7fb7c37fe640 1 -- 172.21.15.32:0/1528502544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8031e00 con 0x55b4ad64de20 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/1528502544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 msgr2=0x55b4ad654d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8004f00 tx=0x7fb7c8031400 comp rx=0 tx=0).stop 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/1528502544 shutdown_connections 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad654d80 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/1528502544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad6686e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/1528502544 >> 172.21.15.32:0/1528502544 conn(0x55b4ad649b30 msgr2=0x55b4ad654670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/1528502544 shutdown_connections 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/1528502544 wait complete. 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 -- start start 2024-04-23T18:22:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.504+0000 7fb7e2b9c740 1 Processor -- start 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad664900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81a9fb0 con 0x55b4ad64de20 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81ab420 con 0x55b4ad65cc10 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d81aa680 con 0x55b4ad64d500 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7e2b9c740 10 client.0 _ll_register_callbacks cb 0x7fb7c802aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45870/0 (socket says 172.21.15.32:45870) 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3701912444 learned_addr learned my addr 172.21.15.32:0/3701912444 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad664900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3701912444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 msgr2=0x55b4ad664900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad664900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 msgr2=0x55b4ad65fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7de34c640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 -- 172.21.15.32:0/3701912444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4ad660500 con 0x55b4ad64de20 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7de34c640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7dd34a640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009340 tx=0x7fb7c80319e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6aeb522b555a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.505+0000 7fb7c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c805a040 con 0x55b4ad64de20 2024-04-23T18:22:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fb7c8004a70 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7c8005440 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb7c803b460 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 10 client.16440 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b4ad6d2800 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8005440 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fb7c8054040 con 0x55b4ad64de20 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 10 client.16440 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 20 client.16440 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad6d2e40 0x55b4ad6d5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 4 client.16440 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4ad65a090 con 0x55b4ad6d2e40 2024-04-23T18:22:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.506+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad6d2e40 0x55b4ad6d5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.507+0000 7fb7ddb4b640 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad6d2e40 0x55b4ad6d5260 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7838f3a3e5251cf8 server_cookie=901a71c22fdd876f in_seq=0 out_seq=0 2024-04-23T18:22:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.507+0000 7fb7c17fa640 10 client.16440 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7c17fa640 1 -- 172.21.15.32:0/3701912444 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4ad65a090 con 0x55b4ad6d2e40 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7c17fa640 10 client.16440 handle_command_reply: tid=0 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 client.16440 shutdown 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 msgr2=0x55b4ad664e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009340 tx=0x7fb7c80319e0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 shutdown_connections 2024-04-23T18:22:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b4ad6d2e40 0x55b4ad6d5260 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4ad65cc10 0x55b4ad65fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4ad64de20 0x55b4ad664e40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7c17fa640 0 client.16440 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 --2- 172.21.15.32:0/3701912444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ad64d500 0x55b4ad664900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.517+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 >> 172.21.15.32:0/3701912444 conn(0x55b4ad649b30 msgr2=0x55b4ad64efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 shutdown_connections 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7e2b9c740 1 -- 172.21.15.32:0/3701912444 wait complete. 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7e2b9c740 20 client.16440 trim_cache size 0 max 16384 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:22:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 msgr2=0x7fb7c40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 0x7fb7c40a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0005fd0 tx=0x7fb7d0009500 comp rx=0 tx=0).stop 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 msgr2=0x7fb7d8197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009970 tx=0x7fb7d4009e30 comp rx=0 tx=0).stop 2024-04-23T18:22:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 shutdown_connections 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7d819bed0 0x7fb7d819c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7d8106170 0x7fb7d8197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb7c40a3d10 0x7fb7c40a61d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 --2- 172.21.15.32:0/4139943095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d81010a0 0x7fb7d8197610 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 >> 172.21.15.32:0/4139943095 conn(0x7fb7d8077f70 msgr2=0x7fb7d80ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 shutdown_connections 2024-04-23T18:22:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:39.518+0000 7fb7dfdaf640 1 -- 172.21.15.32:0/4139943095 wait complete. 2024-04-23T18:22:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.652+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67b8109f30 con 0x7f67b8106f50 2024-04-23T18:22:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.653+0000 7f67b5ffb640 1 -- 172.21.15.32:0/779231699 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11855 (secure 0 0 0) 0x7f67b009cb50 con 0x7f67b8106f50 2024-04-23T18:22:39.647 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:22:39.647 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:21:55.867165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:22:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 msgr2=0x7f67a80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 0x7f67a80a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005fd0 tx=0x7f67ac00b390 comp rx=0 tx=0).stop 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 msgr2=0x7f67b8101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f67b000cad0 tx=0x7f67b000cf90 comp rx=0 tx=0).stop 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 shutdown_connections 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f67a80a3d20 0x7f67a80a61e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67b8106f50 0x7f67b8101bf0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8069c10 0x7f67b81016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 --2- 172.21.15.32:0/779231699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b8069250 0x7f67b8101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 >> 172.21.15.32:0/779231699 conn(0x7f67b8077c40 msgr2=0x7f67b80fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 shutdown_connections 2024-04-23T18:22:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.655+0000 7f67bf01e640 1 -- 172.21.15.32:0/779231699 wait complete. 2024-04-23T18:22:39.659 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id e434819a --release --await' 2024-04-23T18:22:39.660 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id e434819a --release --await 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 -- 172.21.15.32:0/269051930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c074500 msgr2=0x7f040c0748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 --2- 172.21.15.32:0/269051930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c074500 0x7f040c0748e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0408009900 tx=0x7f040802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 -- 172.21.15.32:0/269051930 shutdown_connections 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 --2- 172.21.15.32:0/269051930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c10e910 0x7f040c110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 --2- 172.21.15.32:0/269051930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074e20 0x7f040c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 --2- 172.21.15.32:0/269051930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c074500 0x7f040c0748e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 -- 172.21.15.32:0/269051930 >> 172.21.15.32:0/269051930 conn(0x7f040c0fda10 msgr2=0x7f040c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 -- 172.21.15.32:0/269051930 shutdown_connections 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.815+0000 7f0412f68640 1 -- 172.21.15.32:0/269051930 wait complete. 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 Processor -- start 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 -- start start 2024-04-23T18:22:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 0x7f040c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c074e20 0x7f040c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f040c114490 con 0x7f040c10e910 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f040c114310 con 0x7f040c074500 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0412f68640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f040c114610 con 0x7f040c074e20 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45888/0 (socket says 172.21.15.32:45888) 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 -- 172.21.15.32:0/1160797055 learned_addr learned my addr 172.21.15.32:0/1160797055 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0403fff640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c074e20 0x7f040c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0410d04640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 0x7f040c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 -- 172.21.15.32:0/1160797055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c074e20 msgr2=0x7f040c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c074e20 0x7f040c1a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 -- 172.21.15.32:0/1160797055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 msgr2=0x7f040c1a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 0x7f040c1a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0411505640 1 -- 172.21.15.32:0/1160797055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040c19aff0 con 0x7f040c10e910 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.816+0000 7f0410d04640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 0x7f040c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0411505640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f040400eb40 tx=0x7f040400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0401ffb640 1 -- 172.21.15.32:0/1160797055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404004b20 con 0x7f040c10e910 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040c19b2e0 con 0x7f040c10e910 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0401ffb640 1 -- 172.21.15.32:0/1160797055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0404004cc0 con 0x7f040c10e910 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0401ffb640 1 -- 172.21.15.32:0/1160797055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404010720 con 0x7f040c10e910 2024-04-23T18:22:39.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.817+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040c1a7580 con 0x7f040c10e910 2024-04-23T18:22:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.819+0000 7f0401ffb640 1 -- 172.21.15.32:0/1160797055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0404005560 con 0x7f040c10e910 2024-04-23T18:22:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.819+0000 7f0415922740 1 Processor -- start 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.819+0000 7f0415922740 1 -- start start 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e4fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e506f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0401ffb640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 0x7f03f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a29e4616d0 con 0x55a29e507480 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a29e3fe1a0 con 0x55a29e4fc490 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0415922740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a29e45d690 con 0x55a29e4fcdb0 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0410d04640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 0x7f03f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45892/0 (socket says 172.21.15.32:45892) 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 -- 172.21.15.32:0/2491683339 learned_addr learned my addr 172.21.15.32:0/2491683339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0401ffb640 1 -- 172.21.15.32:0/1160797055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f04040d0f70 con 0x7f040c10e910 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0403fff640 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e506f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0403fff640 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e4fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 -- 172.21.15.32:0/2491683339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 msgr2=0x55a29e506f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e506f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 -- 172.21.15.32:0/2491683339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 msgr2=0x55a29e4fc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e4fc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 -- 172.21.15.32:0/2491683339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a29e4085b0 con 0x55a29e507480 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0411505640 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f040409cec0 tx=0x7f040409ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0aa872634a18b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f0410d04640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 0x7f03f40a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0408000c00 tx=0x7f0408031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.820+0000 7f03f2ffd640 1 -- 172.21.15.32:0/2491683339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040409d630 con 0x55a29e507480 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f03f2ffd640 1 -- 172.21.15.32:0/2491683339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04040a10d0 con 0x55a29e507480 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f03f2ffd640 1 -- 172.21.15.32:0/2491683339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040409e490 con 0x55a29e507480 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 -- 172.21.15.32:0/2491683339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 msgr2=0x55a29e509870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f040409cec0 tx=0x7f040409ca70 comp rx=0 tx=0).stop 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 -- 172.21.15.32:0/2491683339 shutdown_connections 2024-04-23T18:22:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e509870 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e506f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 --2- 172.21.15.32:0/2491683339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e4fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 -- 172.21.15.32:0/2491683339 >> 172.21.15.32:0/2491683339 conn(0x55a29e4f87b0 msgr2=0x55a29e4f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 -- 172.21.15.32:0/2491683339 shutdown_connections 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.821+0000 7f0415922740 1 -- 172.21.15.32:0/2491683339 wait complete. 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 -- start start 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 Processor -- start 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e506f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e504270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a29e50a250 con 0x55a29e507480 2024-04-23T18:22:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a29e3fbb20 con 0x55a29e4fc490 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a29e50a3d0 con 0x55a29e4fcdb0 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0415922740 10 client.0 _ll_register_callbacks cb 0x55a29e3fb530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45904/0 (socket says 172.21.15.32:45904) 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 -- 172.21.15.32:0/176033567 learned_addr learned my addr 172.21.15.32:0/176033567 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0403fff640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e504270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0410d04640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e506f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 -- 172.21.15.32:0/176033567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 msgr2=0x55a29e504270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e504270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 -- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 msgr2=0x55a29e506f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e506f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.822+0000 7f0411505640 1 -- 172.21.15.32:0/176033567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a29e173920 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0411505640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f04040cece0 tx=0x7f040409cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0830725928b0396 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04040b5020 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f040407da50 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04040a10d0 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f04040a12e0 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 10 client.16452 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a29e23fd30 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04040a10d0 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f04040a5710 con 0x55a29e507480 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 10 client.16452 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 10 client.16452 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:39.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 20 client.16452 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e5174d0 0x55a29e5198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 4 client.16452 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.823+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a29e41f500 con 0x55a29e5174d0 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.824+0000 7f0410d04640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e5174d0 0x55a29e5198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.824+0000 7f03ebfff640 10 client.16452 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.824+0000 7f0410d04640 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e5174d0 0x55a29e5198c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db83b845db86f20 server_cookie=fbec401d371d7593 in_seq=0 out_seq=0 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f03ebfff640 1 -- 172.21.15.32:0/176033567 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a29e41f500 con 0x55a29e5174d0 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f03ebfff640 10 client.16452 handle_command_reply: tid=0 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 client.16452 shutdown 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 msgr2=0x55a29e504960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f04040cece0 tx=0x7f040409cf30 comp rx=0 tx=0).stop 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 shutdown_connections 2024-04-23T18:22:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e5174d0 0x55a29e5198c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e507480 0x55a29e504960 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fcdb0 0x55a29e504270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 --2- 172.21.15.32:0/176033567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e506f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f03ebfff640 0 client.16452 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 >> 172.21.15.32:0/176033567 conn(0x55a29e4f87b0 msgr2=0x55a29e4fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 shutdown_connections 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 1 -- 172.21.15.32:0/176033567 wait complete. 2024-04-23T18:22:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.825+0000 7f0415922740 20 client.16452 trim_cache size 0 max 16384 2024-04-23T18:22:39.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.842+0000 7f0415922740 1 Processor -- start 2024-04-23T18:22:39.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.842+0000 7f0415922740 1 -- start start 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.842+0000 7f0415922740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e503ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.842+0000 7f0415922740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0415922740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e507480 0x55a29e501a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0415922740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1b2620 con 0x55a29e4fcdb0 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0415922740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1afe60 con 0x55a29e4fc490 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0415922740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1adcd0 con 0x55a29e507480 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45906/0 (socket says 172.21.15.32:45906) 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 -- 172.21.15.32:0/761434671 learned_addr learned my addr 172.21.15.32:0/761434671 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0411505640 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e507480 0x55a29e501a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 -- 172.21.15.32:0/761434671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e507480 msgr2=0x55a29e501a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e507480 0x55a29e501a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 -- 172.21.15.32:0/761434671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 msgr2=0x55a29e503ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e503ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 -- 172.21.15.32:0/761434671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a29e503220 con 0x55a29e4fcdb0 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f0403fff640 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f040c19c060 tx=0x7f03fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9f97a746bcfb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f03f27fc640 1 -- 172.21.15.32:0/761434671 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc005220 con 0x55a29e4fcdb0 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f03f27fc640 1 -- 172.21.15.32:0/761434671 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03fc004c50 con 0x55a29e4fcdb0 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.843+0000 7f03f27fc640 1 -- 172.21.15.32:0/761434671 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc010970 con 0x55a29e4fcdb0 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- 172.21.15.32:0/761434671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 msgr2=0x55a29e5014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f040c19c060 tx=0x7f03fc004730 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- 172.21.15.32:0/761434671 shutdown_connections 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e507480 0x55a29e501a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e5014d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 --2- 172.21.15.32:0/761434671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e4fc490 0x55a29e503ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- 172.21.15.32:0/761434671 >> 172.21.15.32:0/761434671 conn(0x55a29e4f87b0 msgr2=0x55a29e5037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- 172.21.15.32:0/761434671 shutdown_connections 2024-04-23T18:22:39.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- 172.21.15.32:0/761434671 wait complete. 2024-04-23T18:22:39.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 -- start start 2024-04-23T18:22:39.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.844+0000 7f0415922740 1 Processor -- start 2024-04-23T18:22:39.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fc490 0x55a29e518be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 0x55a29e5766e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1ae320 con 0x55a29e4fcdb0 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1af790 con 0x55a29e507480 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1ae9f0 con 0x55a29e4fc490 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 10 client.0 _ll_register_callbacks cb 0x7f04040a6a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45922/0 (socket says 172.21.15.32:45922) 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 -- 172.21.15.32:0/811302552 learned_addr learned my addr 172.21.15.32:0/811302552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0410d04640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fc490 0x55a29e518be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0411505640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 0x55a29e5766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 -- 172.21.15.32:0/811302552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fc490 msgr2=0x55a29e518be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fc490 0x55a29e518be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 -- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 msgr2=0x55a29e5766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 0x55a29e5766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 -- 172.21.15.32:0/811302552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a29e576cd0 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0411505640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 0x55a29e5766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0403fff640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f03fc004700 tx=0x7f03fc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff915c1dc22bd2f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f03ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc005230 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f03fc023040 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.845+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03fc004c50 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f03fc035050 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 10 client.16458 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a29e57e0b0 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03fc004c50 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f03fc027070 con 0x55a29e4fcdb0 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 10 client.16458 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 10 client.16458 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 20 client.16458 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e57e820 0x55a29e580c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 4 client.16458 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e434819a", "release": true, "await": true}] 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e434819a", "release": true, "await": true}) -- 0x55a29e46f0f0 con 0x55a29e57e820 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.846+0000 7f0410d04640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e57e820 0x55a29e580c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.847+0000 7f0410d04640 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e57e820 0x55a29e580c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82591178c1b20794 server_cookie=bdb991b54d36cec4 in_seq=0 out_seq=0 2024-04-23T18:22:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.847+0000 7f03ebfff640 10 client.16458 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f03ebfff640 1 -- 172.21.15.32:0/811302552 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a29e46f0f0 con 0x55a29e57e820 2024-04-23T18:22:39.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f03ebfff640 10 client.16458 handle_command_reply: tid=0 2024-04-23T18:22:39.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 client.16458 shutdown 2024-04-23T18:22:39.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 msgr2=0x55a29e519120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f03fc004700 tx=0x7f03fc004a50 comp rx=0 tx=0).stop 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 shutdown_connections 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a29e57e820 0x55a29e580c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a29e507480 0x55a29e5766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a29e4fcdb0 0x55a29e519120 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 --2- 172.21.15.32:0/811302552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a29e4fc490 0x55a29e518be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 >> 172.21.15.32:0/811302552 conn(0x55a29e4f87b0 msgr2=0x55a29e504040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.854+0000 7f03ebfff640 0 client.16458 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 shutdown_connections 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0415922740 1 -- 172.21.15.32:0/811302552 wait complete. 2024-04-23T18:22:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0415922740 20 client.16458 trim_cache size 0 max 16384 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 37, 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 28, 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "e434819a": { 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 28, 2024-04-23T18:22:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:22:39.852 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 msgr2=0x7f03f40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 0x7f03f40a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0408000c00 tx=0x7f0408031400 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 msgr2=0x7f040c19a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f040400eb40 tx=0x7f040400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 shutdown_connections 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f03f40a3d20 0x7f03f40a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f040c10e910 0x7f040c19a870 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f040c074e20 0x7f040c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 --2- 172.21.15.32:0/1160797055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040c074500 0x7f040c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.855+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 >> 172.21.15.32:0/1160797055 conn(0x7f040c0fda10 msgr2=0x7f040c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.856+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 shutdown_connections 2024-04-23T18:22:39.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:22:39.856+0000 7f0412f68640 1 -- 172.21.15.32:0/1160797055 wait complete. 2024-04-23T18:22:39.862 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e434819a', seconds elapsed: 31.7 2024-04-23T18:22:40.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:40 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/779231699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:40.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:40 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/779231699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:40.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:40 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/779231699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:41.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:41 smithi040 ceph-mon[32626]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-23T18:22:41.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:41 smithi064 ceph-mon[33062]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-23T18:22:41.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:41 smithi032 ceph-mon[29221]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-23T18:22:43.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:43 smithi040 ceph-mon[32626]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-04-23T18:22:43.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:43 smithi064 ceph-mon[33062]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-04-23T18:22:43.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:43 smithi032 ceph-mon[29221]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-04-23T18:22:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 435.0 s, 2550.2 tps, lat 273.384 ms stddev 2723.284 2024-04-23T18:22:45.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:45 smithi032 ceph-mon[29221]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 831 KiB/s wr, 68 op/s 2024-04-23T18:22:45.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:45 smithi040 ceph-mon[32626]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 831 KiB/s wr, 68 op/s 2024-04-23T18:22:45.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:45 smithi064 ceph-mon[33062]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 831 KiB/s wr, 68 op/s 2024-04-23T18:22:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:47 smithi040 ceph-mon[32626]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:47.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:47 smithi064 ceph-mon[33062]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:47.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:47 smithi032 ceph-mon[29221]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 440.0 s, 3386.6 tps, lat 29.277 ms stddev 33.108 2024-04-23T18:22:49.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:49 smithi040 ceph-mon[32626]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:49.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:49 smithi064 ceph-mon[33062]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:49.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:49 smithi032 ceph-mon[29221]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 3.1 MiB/s wr, 233 op/s 2024-04-23T18:22:51.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:51 smithi040 ceph-mon[32626]: pgmap v237: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.4 MiB/s wr, 542 op/s 2024-04-23T18:22:51.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:51 smithi064 ceph-mon[33062]: pgmap v237: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.4 MiB/s wr, 542 op/s 2024-04-23T18:22:51.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:51 smithi032 ceph-mon[29221]: pgmap v237: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.4 MiB/s wr, 542 op/s 2024-04-23T18:22:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3050005621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0101a70 msgr2=0x7fbde0105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3050005621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0101a70 0x7fbde0105ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009900 tx=0x7fbdd002a3f0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3050005621 shutdown_connections 2024-04-23T18:22:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3050005621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde0106190 0x7fbde010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3050005621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0101a70 0x7fbde0105ab0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3050005621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde01010c0 0x7fbde01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.643+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3050005621 >> 172.21.15.32:0/3050005621 conn(0x7fbde0077f80 msgr2=0x7fbde0078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.644+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3050005621 shutdown_connections 2024-04-23T18:22:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.644+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3050005621 wait complete. 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.644+0000 7fbde7d8e640 1 Processor -- start 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.644+0000 7fbde7d8e640 1 -- start start 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde0106190 0x7fbde01978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0197e80 0x7fbde019c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01110b0 con 0x7fbde01010c0 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbde0110f30 con 0x7fbde0106190 2024-04-23T18:22:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde7d8e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbde0111230 con 0x7fbde0197e80 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36554/0 (socket says 172.21.15.32:36554) 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3046711220 learned_addr learned my addr 172.21.15.32:0/3046711220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.645+0000 7fbde5329640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde0106190 0x7fbde01978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3046711220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0197e80 msgr2=0x7fbde019c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0197e80 0x7fbde019c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3046711220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde0106190 msgr2=0x7fbde01978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde0106190 0x7fbde01978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3046711220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde019cc40 con 0x7fbde01010c0 2024-04-23T18:22:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00d950 tx=0x7fbddc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbdd6ffd640 1 -- 172.21.15.32:0/3046711220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc017030 con 0x7fbde01010c0 2024-04-23T18:22:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbdd6ffd640 1 -- 172.21.15.32:0/3046711220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbddc004020 con 0x7fbde01010c0 2024-04-23T18:22:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.646+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde019cf30 con 0x7fbde01010c0 2024-04-23T18:22:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.647+0000 7fbdd6ffd640 1 -- 172.21.15.32:0/3046711220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc0047a0 con 0x7fbde01010c0 2024-04-23T18:22:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.647+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde019d450 con 0x7fbde01010c0 2024-04-23T18:22:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.650+0000 7fbdd6ffd640 1 -- 172.21.15.32:0/3046711220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fbddc009a30 con 0x7fbde01010c0 2024-04-23T18:22:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 Processor -- start 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 -- start start 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 0x55a51591ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 0x55a5159272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 0x55a515929bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a515881970 con 0x55a51591cf40 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a51581e4a0 con 0x55a51591c620 2024-04-23T18:22:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdeab80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a51587d930 con 0x55a515927800 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbdd6ffd640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 0x7fbdcc0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 0x55a5159272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbde632b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 0x55a51591ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 0x55a5159272c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36556/0 (socket says 172.21.15.32:36556) 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbde5b2a640 1 -- 172.21.15.32:0/4228352815 learned_addr learned my addr 172.21.15.32:0/4228352815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.651+0000 7fbde5329640 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 0x55a515929bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 0x7fbdcc0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 -- 172.21.15.32:0/4228352815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 msgr2=0x55a515929bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 0x55a515929bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 -- 172.21.15.32:0/4228352815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 msgr2=0x55a5159272c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 0x55a5159272c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 -- 172.21.15.32:0/4228352815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a515594210 con 0x55a51591c620 2024-04-23T18:22:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbdd6ffd640 1 -- 172.21.15.32:0/3046711220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fbddc0d01c0 con 0x7fbde01010c0 2024-04-23T18:22:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde5329640 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 0x55a515929bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 0x55a51591ca00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009fd0 tx=0x7fbdd8012c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f339c4edde7c826c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbde632b640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 0x7fbdcc0a6320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbdd801b000 tx=0x7fbdd801d000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbdcbfff640 1 -- 172.21.15.32:0/4228352815 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd801c7d0 con 0x55a51591c620 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbdcbfff640 1 -- 172.21.15.32:0/4228352815 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdd8020070 con 0x55a51591c620 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.652+0000 7fbdcbfff640 1 -- 172.21.15.32:0/4228352815 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8024b30 con 0x55a51591c620 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.653+0000 7fbdeab80740 1 -- 172.21.15.32:0/4228352815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 msgr2=0x55a51591ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.653+0000 7fbdeab80740 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 0x55a51591ca00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009fd0 tx=0x7fbdd8012c10 comp rx=0 tx=0).stop 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.654+0000 7fbdeab80740 1 -- 172.21.15.32:0/4228352815 shutdown_connections 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.654+0000 7fbdeab80740 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a515927800 0x55a515929bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.654+0000 7fbdeab80740 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591cf40 0x55a5159272c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.654+0000 7fbdeab80740 1 --2- 172.21.15.32:0/4228352815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591c620 0x55a51591ca00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.654+0000 7fbdeab80740 1 -- 172.21.15.32:0/4228352815 >> 172.21.15.32:0/4228352815 conn(0x55a515918b30 msgr2=0x55a515918f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.655+0000 7fbdeab80740 1 -- 172.21.15.32:0/4228352815 shutdown_connections 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.655+0000 7fbdeab80740 1 -- 172.21.15.32:0/4228352815 wait complete. 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.655+0000 7fbdeab80740 1 -- start start 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 Processor -- start 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515924480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515924a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a51592a3d0 con 0x55a515927800 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a51581c0b0 con 0x55a51591cf40 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a51592a550 con 0x55a51591c620 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbdeab80740 10 client.0 _ll_register_callbacks cb 0x55a515809d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbde5b2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35656/0 (socket says 172.21.15.32:35656) 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3640946460 learned_addr learned my addr 172.21.15.32:0/3640946460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5329640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515924480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.656+0000 7fbde632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515924a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 msgr2=0x55a515924480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515924480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3640946460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 msgr2=0x55a515924a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515924a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 -- 172.21.15.32:0/3640946460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a515660620 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbddc004770 tx=0x7fbddc09a690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33faf189f0242882 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbdc97fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc0a0530 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.657+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fbddc0cdbb0 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdd8020070 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbddc0a4df0 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 10 client.26044 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a515936320 con 0x55a51591c620 2024-04-23T18:22:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbddc09c180 con 0x55a51591c620 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fbddc0a4960 con 0x55a51591c620 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 10 client.26044 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 20 client.26044 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a515937790 0x55a515939b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 4 client.26044 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbdd8020070 con 0x55a515937790 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.658+0000 7fbde5329640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a515937790 0x55a515939b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.659+0000 7fbde5329640 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a515937790 0x55a515939b80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95b3227f70719c74 server_cookie=cb2b1504c638c1b0 in_seq=0 out_seq=0 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.659+0000 7fbdc97fa640 10 client.26044 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.660+0000 7fbdc97fa640 1 -- 172.21.15.32:0/3640946460 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbdd8020070 con 0x55a515937790 2024-04-23T18:22:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.660+0000 7fbdc97fa640 10 client.26044 handle_command_reply: tid=0 2024-04-23T18:22:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.660+0000 7fbdeab80740 1 client.26044 shutdown 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.661+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 msgr2=0x55a515927190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.661+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbddc004770 tx=0x7fbddc09a690 comp rx=0 tx=0).stop 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 shutdown_connections 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a515937790 0x55a515939b80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515924a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515924480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 --2- 172.21.15.32:0/3640946460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515927190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 >> 172.21.15.32:0/3640946460 conn(0x55a515918b30 msgr2=0x55a51591e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.662+0000 7fbdc97fa640 0 client.26044 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.663+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 shutdown_connections 2024-04-23T18:22:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.663+0000 7fbdeab80740 1 -- 172.21.15.32:0/3640946460 wait complete. 2024-04-23T18:22:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.663+0000 7fbdeab80740 20 client.26044 trim_cache size 0 max 16384 2024-04-23T18:22:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.691+0000 7fbdeab80740 1 Processor -- start 2024-04-23T18:22:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.691+0000 7fbdeab80740 1 -- start start 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515933290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515917b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01bf240 con 0x55a515927800 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01a5a40 con 0x55a51591cf40 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbdeab80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01a38b0 con 0x55a51591c620 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbde632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbde632b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36574/0 (socket says 172.21.15.32:36574) 2024-04-23T18:22:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.692+0000 7fbde632b640 1 -- 172.21.15.32:0/1485000232 learned_addr learned my addr 172.21.15.32:0/1485000232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 -- 172.21.15.32:0/1485000232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 msgr2=0x55a515933290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:22:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515933290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 -- 172.21.15.32:0/1485000232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 msgr2=0x55a515917b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:22:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515917b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 -- 172.21.15.32:0/1485000232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a51591cd70 con 0x55a515927800 2024-04-23T18:22:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.693+0000 7fbde632b640 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80246b0 tx=0x7fbdd80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcfe2ba0b207bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.694+0000 7fbdcb7fe640 1 -- 172.21.15.32:0/1485000232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8024410 con 0x55a515927800 2024-04-23T18:22:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.694+0000 7fbdcb7fe640 1 -- 172.21.15.32:0/1485000232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdd8033070 con 0x55a515927800 2024-04-23T18:22:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.694+0000 7fbdcb7fe640 1 -- 172.21.15.32:0/1485000232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8024780 con 0x55a515927800 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 -- 172.21.15.32:0/1485000232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 msgr2=0x55a515926cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80246b0 tx=0x7fbdd80094c0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 -- 172.21.15.32:0/1485000232 shutdown_connections 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a515927800 0x55a515926cc0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a51591cf40 0x55a515917b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 --2- 172.21.15.32:0/1485000232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591c620 0x55a515933290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.695+0000 7fbdeab80740 1 -- 172.21.15.32:0/1485000232 >> 172.21.15.32:0/1485000232 conn(0x55a515918b30 msgr2=0x55a515917420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.698+0000 7fbdeab80740 1 -- 172.21.15.32:0/1485000232 shutdown_connections 2024-04-23T18:22:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.699+0000 7fbdeab80740 1 -- 172.21.15.32:0/1485000232 wait complete. 2024-04-23T18:22:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.700+0000 7fbdeab80740 1 -- start start 2024-04-23T18:22:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.700+0000 7fbdeab80740 1 Processor -- start 2024-04-23T18:22:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.703+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591c620 0x55a51592fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.703+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.704+0000 7fbdeab80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a515927800 0x55a5159212d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.704+0000 7fbdeab80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01a3f00 con 0x55a51591c620 2024-04-23T18:22:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.704+0000 7fbdeab80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01a5370 con 0x55a515927800 2024-04-23T18:22:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.704+0000 7fbdeab80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbde01a45d0 con 0x55a51591cf40 2024-04-23T18:22:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.704+0000 7fbdeab80740 10 client.0 _ll_register_callbacks cb 0x7fbddc09bcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.705+0000 7fbde5329640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.706+0000 7fbde5329640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35658/0 (socket says 172.21.15.32:35658) 2024-04-23T18:22:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.706+0000 7fbde5329640 1 -- 172.21.15.32:0/818798687 learned_addr learned my addr 172.21.15.32:0/818798687 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:22:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.707+0000 7fbde5329640 1 -- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a515927800 msgr2=0x55a5159212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.707+0000 7fbde5329640 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a515927800 0x55a5159212d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.707+0000 7fbde5329640 1 -- 172.21.15.32:0/818798687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591c620 msgr2=0x55a51592fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.707+0000 7fbde5329640 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591c620 0x55a51592fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.707+0000 7fbde5329640 1 -- 172.21.15.32:0/818798687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a515921990 con 0x55a51591cf40 2024-04-23T18:22:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.708+0000 7fbde5329640 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0031440 tx=0x7fbdd00041a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32fb82957087bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.711+0000 7fbdc97fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:22:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.711+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd004a060 con 0x55a51591cf40 2024-04-23T18:22:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.711+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7fbdd0045dc0 con 0x55a51591cf40 2024-04-23T18:22:52.706 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.711+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdd8033070 con 0x55a51591cf40 2024-04-23T18:22:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.715+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbdd0035040 con 0x55a51591cf40 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.716+0000 7fbdeab80740 10 client.26047 fetch_fsmap learned FSMap version 37 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.716+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a51599dc20 con 0x55a51591cf40 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdd8033070 con 0x55a51591cf40 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fbdd00046c0 con 0x55a51591cf40 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 10 client.26047 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 10 client.26047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 20 client.26047 populate_metadata read hostname 'smithi032' 2024-04-23T18:22:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a51599e310 0x55a5159a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 4 client.26047 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.717+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a51588f390 con 0x55a51599e310 2024-04-23T18:22:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.719+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a51599e310 0x55a5159a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.722+0000 7fbde5b2a640 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a51599e310 0x55a5159a0730 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba3eeeb60a9045c server_cookie=5cf590c588124f8a in_seq=0 out_seq=0 2024-04-23T18:22:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.723+0000 7fbdc97fa640 10 client.26047 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.745+0000 7fbdc97fa640 1 -- 172.21.15.32:0/818798687 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a51588f390 con 0x55a51599e310 2024-04-23T18:22:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.745+0000 7fbdc97fa640 10 client.26047 handle_command_reply: tid=0 2024-04-23T18:22:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.745+0000 7fbdeab80740 1 client.26047 shutdown 2024-04-23T18:22:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 msgr2=0x55a515920d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0031440 tx=0x7fbdd00041a0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 shutdown_connections 2024-04-23T18:22:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a51599e310 0x55a5159a0730 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a515927800 0x55a5159212d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a51591cf40 0x55a515920d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdc97fa640 0 client.26047 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 --2- 172.21.15.32:0/818798687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a51591c620 0x55a51592fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.747+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 >> 172.21.15.32:0/818798687 conn(0x55a515918b30 msgr2=0x55a515917420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.748+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 shutdown_connections 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.748+0000 7fbdeab80740 1 -- 172.21.15.32:0/818798687 wait complete. 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.748+0000 7fbdeab80740 20 client.26047 trim_cache size 0 max 16384 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:22:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:22:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:22:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 msgr2=0x7fbdcc0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 0x7fbdcc0a6320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbdd801b000 tx=0x7fbdd801d000 comp rx=0 tx=0).stop 2024-04-23T18:22:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 msgr2=0x7fbde0197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00d950 tx=0x7fbddc00de10 comp rx=0 tx=0).stop 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 shutdown_connections 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbde0197e80 0x7fbde019c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbde0106190 0x7fbde01978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbdcc0a3e60 0x7fbdcc0a6320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 --2- 172.21.15.32:0/3046711220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde01010c0 0x7fbde0197360 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 >> 172.21.15.32:0/3046711220 conn(0x7fbde0077f80 msgr2=0x7fbde00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 shutdown_connections 2024-04-23T18:22:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:22:52.749+0000 7fbde7d8e640 1 -- 172.21.15.32:0/3046711220 wait complete. 2024-04-23T18:22:53.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:53 smithi040 ceph-mon[32626]: pgmap v238: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 7.4 MiB/s wr, 538 op/s 2024-04-23T18:22:53.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:53 smithi064 ceph-mon[33062]: pgmap v238: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 7.4 MiB/s wr, 538 op/s 2024-04-23T18:22:53.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:53 smithi032 ceph-mon[29221]: pgmap v238: 97 pgs: 97 active+clean; 3.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 7.4 MiB/s wr, 538 op/s 2024-04-23T18:22:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 445.0 s, 3011.6 tps, lat 33.355 ms stddev 36.053 2024-04-23T18:22:55.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:55 smithi040 ceph-mon[32626]: pgmap v239: 97 pgs: 97 active+clean; 3.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.4 MiB/s wr, 618 op/s 2024-04-23T18:22:55.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:55 smithi064 ceph-mon[33062]: pgmap v239: 97 pgs: 97 active+clean; 3.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.4 MiB/s wr, 618 op/s 2024-04-23T18:22:55.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:55 smithi032 ceph-mon[29221]: pgmap v239: 97 pgs: 97 active+clean; 3.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.4 MiB/s wr, 618 op/s 2024-04-23T18:22:57.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:57 smithi040 ceph-mon[32626]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-23T18:22:57.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:57 smithi064 ceph-mon[33062]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-23T18:22:57.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:57 smithi032 ceph-mon[29221]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-23T18:22:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 450.0 s, 3281.0 tps, lat 30.602 ms stddev 31.675 2024-04-23T18:22:59.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:22:59 smithi040 ceph-mon[32626]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 7.9 MiB/s wr, 616 op/s 2024-04-23T18:22:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:22:59 smithi064 ceph-mon[33062]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 7.9 MiB/s wr, 616 op/s 2024-04-23T18:22:59.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:22:59 smithi032 ceph-mon[29221]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 7.9 MiB/s wr, 616 op/s 2024-04-23T18:23:01.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:01 smithi040 ceph-mon[32626]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-04-23T18:23:01.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:01 smithi064 ceph-mon[33062]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-04-23T18:23:01.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:01 smithi032 ceph-mon[29221]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-04-23T18:23:03.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:03 smithi040 ceph-mon[32626]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.5 MiB/s wr, 600 op/s 2024-04-23T18:23:03.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:03 smithi064 ceph-mon[33062]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.5 MiB/s wr, 600 op/s 2024-04-23T18:23:03.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:03 smithi032 ceph-mon[29221]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.5 MiB/s wr, 600 op/s 2024-04-23T18:23:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 455.0 s, 1353.8 tps, lat 73.027 ms stddev 200.500 2024-04-23T18:23:05.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:05 smithi040 ceph-mon[32626]: pgmap v244: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 15 MiB/s wr, 1k op/s 2024-04-23T18:23:05.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:05 smithi064 ceph-mon[33062]: pgmap v244: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 15 MiB/s wr, 1k op/s 2024-04-23T18:23:05.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:05 smithi032 ceph-mon[29221]: pgmap v244: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 15 MiB/s wr, 1k op/s 2024-04-23T18:23:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 -- 172.21.15.32:0/4059883359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e301150a0 msgr2=0x7f9e30117490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 --2- 172.21.15.32:0/4059883359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e301150a0 0x7f9e30117490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009ca0 tx=0x7f9e2802a950 comp rx=0 tx=0).stop 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 -- 172.21.15.32:0/4059883359 shutdown_connections 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 --2- 172.21.15.32:0/4059883359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e301150a0 0x7f9e30117490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 --2- 172.21.15.32:0/4059883359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e30109a80 0x7f9e30114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 --2- 172.21.15.32:0/4059883359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109160 0x7f9e30109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 -- 172.21.15.32:0/4059883359 >> 172.21.15.32:0/4059883359 conn(0x7f9e3006d030 msgr2=0x7f9e3006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 -- 172.21.15.32:0/4059883359 shutdown_connections 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 -- 172.21.15.32:0/4059883359 wait complete. 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.891+0000 7f9e35c72640 1 Processor -- start 2024-04-23T18:23:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 -- start start 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 0x7f9e301a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109a80 0x7f9e301a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3011aa60 con 0x7f9e301150a0 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3011a8e0 con 0x7f9e30109160 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e35c72640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3011abe0 con 0x7f9e30109a80 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59456/0 (socket says 172.21.15.32:59456) 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 0x7f9e301a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 -- 172.21.15.32:0/2614279852 learned_addr learned my addr 172.21.15.32:0/2614279852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e27fff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109a80 0x7f9e301a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 -- 172.21.15.32:0/2614279852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109a80 msgr2=0x7f9e301a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109a80 0x7f9e301a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 -- 172.21.15.32:0/2614279852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 msgr2=0x7f9e301a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 0x7f9e301a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 -- 172.21.15.32:0/2614279852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e301a81d0 con 0x7f9e301150a0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 0x7f9e301a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.892+0000 7f9e2ffff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9e28005e30 tx=0x7f9e28003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.893+0000 7f9e2d7fa640 1 -- 172.21.15.32:0/2614279852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28034040 con 0x7f9e301150a0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.893+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e301a8460 con 0x7f9e301150a0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.893+0000 7f9e2d7fa640 1 -- 172.21.15.32:0/2614279852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e28003e50 con 0x7f9e301150a0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.893+0000 7f9e2d7fa640 1 -- 172.21.15.32:0/2614279852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28004630 con 0x7f9e301150a0 2024-04-23T18:23:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.893+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3010cb00 con 0x7f9e301150a0 2024-04-23T18:23:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.897+0000 7f9e38986740 1 Processor -- start 2024-04-23T18:23:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.897+0000 7f9e2d7fa640 1 -- 172.21.15.32:0/2614279852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9e28037020 con 0x7f9e301150a0 2024-04-23T18:23:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 -- start start 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b3fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b4a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560e39aa4970 con 0x560e39b3ff40 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560e39a414a0 con 0x560e39b4a800 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e38986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560e39aa0930 con 0x560e39b3f620 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e2d7fa640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 0x7f9e1c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:53748/0 (socket says 172.21.15.32:53748) 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e27fff640 1 -- 172.21.15.32:0/887466738 learned_addr learned my addr 172.21.15.32:0/887466738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b4a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.898+0000 7f9e27fff640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b3fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e2ffff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 0x7f9e1c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e2d7fa640 1 -- 172.21.15.32:0/2614279852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f9e280f1520 con 0x7f9e301150a0 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 -- 172.21.15.32:0/887466738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 msgr2=0x560e39b3fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b3fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 -- 172.21.15.32:0/887466738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 msgr2=0x560e39b4a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b4a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 -- 172.21.15.32:0/887466738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e397b7210 con 0x560e39b4a800 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b4a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e27fff640 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e18005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0b173bd41bebe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e25ffb640 1 -- 172.21.15.32:0/887466738 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1800bb50 con 0x560e39b4a800 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e2ffff640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 0x7f9e1c0a6250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e280b9f60 tx=0x7f9e28087c40 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e25ffb640 1 -- 172.21.15.32:0/887466738 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e1800bcf0 con 0x560e39b4a800 2024-04-23T18:23:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.899+0000 7f9e25ffb640 1 -- 172.21.15.32:0/887466738 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e180055c0 con 0x560e39b4a800 2024-04-23T18:23:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.900+0000 7f9e38986740 1 -- 172.21.15.32:0/887466738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 msgr2=0x560e39b4cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.900+0000 7f9e38986740 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e18005f90 comp rx=0 tx=0).stop 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- 172.21.15.32:0/887466738 shutdown_connections 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b4cbf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b4a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- 172.21.15.32:0/887466738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b3fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- 172.21.15.32:0/887466738 >> 172.21.15.32:0/887466738 conn(0x560e39b3bb30 msgr2=0x560e39b3bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- 172.21.15.32:0/887466738 shutdown_connections 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- 172.21.15.32:0/887466738 wait complete. 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- start start 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 Processor -- start 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b4a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b47920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560e39b4d430 con 0x560e39b3ff40 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560e39a3f0b0 con 0x560e39b4a800 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.901+0000 7f9e38986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560e39b4d5b0 con 0x560e39b3f620 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e38986740 10 client.0 _ll_register_callbacks cb 0x560e39a2cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59482/0 (socket says 172.21.15.32:59482) 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 -- 172.21.15.32:0/3230984169 learned_addr learned my addr 172.21.15.32:0/3230984169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e27fff640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b47920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e27fff640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b4a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 -- 172.21.15.32:0/3230984169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 msgr2=0x560e39b4a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b4a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 -- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 msgr2=0x560e39b47920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b47920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 -- 172.21.15.32:0/3230984169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e39883620 con 0x560e39b3ff40 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e27fff640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b47920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9e20010f10 tx=0x7f9e2000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d20c565d07df9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e0effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.902+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20005020 con 0x560e39b3ff40 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.903+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f9e200051c0 con 0x560e39b3ff40 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.903+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e1800bcf0 con 0x560e39b3ff40 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.903+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e20005400 con 0x560e39b3ff40 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.903+0000 7f9e38986740 10 client.16485 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.903+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560e39b593a0 con 0x560e39b3ff40 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.904+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e1800bcf0 con 0x560e39b3ff40 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.904+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f9e20020020 con 0x560e39b3ff40 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 10 client.16485 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 10 client.16485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 20 client.16485 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39b5a8b0 0x560e39b5cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 4 client.16485 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e39ab2390 con 0x560e39b5a8b0 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.905+0000 7f9e27fff640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39b5a8b0 0x560e39b5cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.906+0000 7f9e27fff640 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39b5a8b0 0x560e39b5cca0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cd99936b04fdb55 server_cookie=de47d447826767a9 in_seq=0 out_seq=0 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.906+0000 7f9e0effd640 10 client.16485 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e0effd640 1 -- 172.21.15.32:0/3230984169 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e39ab2390 con 0x560e39b5a8b0 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e0effd640 10 client.16485 handle_command_reply: tid=0 2024-04-23T18:23:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 client.16485 shutdown 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 msgr2=0x560e39b473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9e20010f10 tx=0x7f9e2000f7f0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 shutdown_connections 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39b5a8b0 0x560e39b5cca0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b4a800 0x560e39b47920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b3ff40 0x560e39b473e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 --2- 172.21.15.32:0/3230984169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b4a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 >> 172.21.15.32:0/3230984169 conn(0x560e39b3bb30 msgr2=0x560e39b418b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.907+0000 7f9e0effd640 0 client.16485 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.908+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 shutdown_connections 2024-04-23T18:23:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.908+0000 7f9e38986740 1 -- 172.21.15.32:0/3230984169 wait complete. 2024-04-23T18:23:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.908+0000 7f9e38986740 20 client.16485 trim_cache size 0 max 16384 2024-04-23T18:23:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.931+0000 7f9e38986740 1 Processor -- start 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.931+0000 7f9e38986740 1 -- start start 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.931+0000 7f9e38986740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.931+0000 7f9e38986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3ff40 0x560e39b49c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e38986740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b62570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e38986740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b6b20 con 0x560e39b4a800 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e38986740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b4360 con 0x560e39b3f620 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e38986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b21f0 con 0x560e39b3ff40 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:53768/0 (socket says 172.21.15.32:53768) 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 -- 172.21.15.32:0/1530378765 learned_addr learned my addr 172.21.15.32:0/1530378765 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b62570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3ff40 0x560e39b49c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 -- 172.21.15.32:0/1530378765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3ff40 msgr2=0x560e39b49c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3ff40 0x560e39b49c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 -- 172.21.15.32:0/1530378765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 msgr2=0x560e39b62570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b62570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 -- 172.21.15.32:0/1530378765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e39b372e0 con 0x560e39b3f620 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e27fff640 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009ba0 tx=0x7f9e18005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42889bff2386b33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e257fa640 1 -- 172.21.15.32:0/1530378765 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18005610 con 0x560e39b3f620 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e257fa640 1 -- 172.21.15.32:0/1530378765 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e180358f0 con 0x560e39b3f620 2024-04-23T18:23:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.932+0000 7f9e257fa640 1 -- 172.21.15.32:0/1530378765 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18003df0 con 0x560e39b3f620 2024-04-23T18:23:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 -- 172.21.15.32:0/1530378765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 msgr2=0x560e39b562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009ba0 tx=0x7f9e18005f90 comp rx=0 tx=0).stop 2024-04-23T18:23:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 -- 172.21.15.32:0/1530378765 shutdown_connections 2024-04-23T18:23:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b62570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3ff40 0x560e39b49c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 --2- 172.21.15.32:0/1530378765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3f620 0x560e39b562f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 -- 172.21.15.32:0/1530378765 >> 172.21.15.32:0/1530378765 conn(0x560e39b3bb30 msgr2=0x560e39b3acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 -- 172.21.15.32:0/1530378765 shutdown_connections 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.933+0000 7f9e38986740 1 -- 172.21.15.32:0/1530378765 wait complete. 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 -- start start 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 Processor -- start 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3ff40 0x560e39b43c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b44150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.934+0000 7f9e38986740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b2840 con 0x560e39b4a800 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e38986740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b3c90 con 0x560e39b3ff40 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e38986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301b2ef0 con 0x560e39b3f620 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e38986740 10 client.0 _ll_register_callbacks cb 0x7f9e200107d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:35378/0 (socket says 172.21.15.32:35378) 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 -- 172.21.15.32:0/2737247311 learned_addr learned my addr 172.21.15.32:0/2737247311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3ff40 0x560e39b43c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b44150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 -- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3ff40 msgr2=0x560e39b43c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3ff40 0x560e39b43c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 -- 172.21.15.32:0/2737247311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 msgr2=0x560e39b44150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b44150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 -- 172.21.15.32:0/2737247311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e39b44690 con 0x560e39b3f620 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e27fff640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e1800bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e106ea7f9602c050 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.935+0000 7f9e0effd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.936+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e180037f0 con 0x560e39b3f620 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.936+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e180358f0 con 0x560e39b3f620 2024-04-23T18:23:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.936+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8484+0+0 (secure 0 0 0) 0x7f9e18003990 con 0x560e39b3f620 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.938+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e1804a050 con 0x560e39b3f620 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 10 client.26062 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560e39bc0ab0 con 0x560e39b3f620 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e180358f0 con 0x560e39b3f620 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f9e18023040 con 0x560e39b3f620 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 10 client.26062 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 10 client.26062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 20 client.26062 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39bc0fc0 0x560e39bc33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 4 client.26062 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.939+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e39aac200 con 0x560e39bc0fc0 2024-04-23T18:23:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.940+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39bc0fc0 0x560e39bc33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.940+0000 7f9e2f7fe640 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39bc0fc0 0x560e39bc33e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db24b2f72033d352 server_cookie=882ab0e9b4e0f736 in_seq=0 out_seq=0 2024-04-23T18:23:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.940+0000 7f9e0effd640 10 client.26062 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.958+0000 7f9e0effd640 1 -- 172.21.15.32:0/2737247311 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e39aac200 con 0x560e39bc0fc0 2024-04-23T18:23:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.958+0000 7f9e0effd640 10 client.26062 handle_command_reply: tid=0 2024-04-23T18:23:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.958+0000 7f9e38986740 1 client.26062 shutdown 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 msgr2=0x560e39b529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e1800bc70 comp rx=0 tx=0).stop 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 shutdown_connections 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e39bc0fc0 0x560e39bc33e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e39b4a800 0x560e39b44150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e0effd640 0 client.26062 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e39b3ff40 0x560e39b43c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 --2- 172.21.15.32:0/2737247311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e39b3f620 0x560e39b529c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.959+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 >> 172.21.15.32:0/2737247311 conn(0x560e39b3bb30 msgr2=0x560e39b46c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 shutdown_connections 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e38986740 1 -- 172.21.15.32:0/2737247311 wait complete. 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e38986740 20 client.26062 trim_cache size 0 max 16384 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:23:05.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:23:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:23:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 msgr2=0x7f9e1c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 0x7f9e1c0a6250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e280b9f60 tx=0x7f9e28087c40 comp rx=0 tx=0).stop 2024-04-23T18:23:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 msgr2=0x7f9e301a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9e28005e30 tx=0x7f9e28003ce0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 shutdown_connections 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e1c0a3d90 0x7f9e1c0a6250 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e301150a0 0x7f9e301a7aa0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e30109a80 0x7f9e301a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 --2- 172.21.15.32:0/2614279852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30109160 0x7f9e301a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.960+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 >> 172.21.15.32:0/2614279852 conn(0x7f9e3006d030 msgr2=0x7f9e3010a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.961+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 shutdown_connections 2024-04-23T18:23:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:05.961+0000 7f9e35c72640 1 -- 172.21.15.32:0/2614279852 wait complete. 2024-04-23T18:23:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:07 smithi040 ceph-mon[32626]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-04-23T18:23:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:07 smithi040 ceph-mon[32626]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:23:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:07 smithi064 ceph-mon[33062]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-04-23T18:23:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:07 smithi064 ceph-mon[33062]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:23:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:07 smithi032 ceph-mon[29221]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-04-23T18:23:07.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:07 smithi032 ceph-mon[29221]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:23:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 460.0 s, 2541.6 tps, lat 39.720 ms stddev 48.090 2024-04-23T18:23:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:09 smithi040 ceph-mon[32626]: pgmap v247: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 MiB/s wr, 1.91k op/s 2024-04-23T18:23:09.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:09 smithi064 ceph-mon[33062]: pgmap v247: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 MiB/s wr, 1.91k op/s 2024-04-23T18:23:09.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:09 smithi032 ceph-mon[29221]: pgmap v247: 97 pgs: 97 active+clean; 4.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 MiB/s wr, 1.91k op/s 2024-04-23T18:23:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:11 smithi040 ceph-mon[32626]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 40 MiB/s wr, 1.90k op/s 2024-04-23T18:23:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:11 smithi040 ceph-mon[32626]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:23:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:11 smithi064 ceph-mon[33062]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 40 MiB/s wr, 1.90k op/s 2024-04-23T18:23:11.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:11 smithi064 ceph-mon[33062]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:23:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:11 smithi032 ceph-mon[29221]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 40 MiB/s wr, 1.90k op/s 2024-04-23T18:23:11.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:11 smithi032 ceph-mon[29221]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:23:13.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:13 smithi040 ceph-mon[32626]: pgmap v250: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 MiB/s wr, 1.77k op/s 2024-04-23T18:23:13.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:13 smithi064 ceph-mon[33062]: pgmap v250: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 MiB/s wr, 1.77k op/s 2024-04-23T18:23:13.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:13 smithi032 ceph-mon[29221]: pgmap v250: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 MiB/s wr, 1.77k op/s 2024-04-23T18:23:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 465.0 s, 2769.4 tps, lat 36.062 ms stddev 39.092 2024-04-23T18:23:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:15 smithi040 ceph-mon[32626]: pgmap v251: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.0 MiB/s wr, 666 op/s 2024-04-23T18:23:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:15 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:15 smithi064 ceph-mon[33062]: pgmap v251: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.0 MiB/s wr, 666 op/s 2024-04-23T18:23:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:15 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:15.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:15 smithi032 ceph-mon[29221]: pgmap v251: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.0 MiB/s wr, 666 op/s 2024-04-23T18:23:15.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:15 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:17 smithi040 ceph-mon[32626]: pgmap v252: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.9 MiB/s wr, 939 op/s 2024-04-23T18:23:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:17 smithi064 ceph-mon[33062]: pgmap v252: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.9 MiB/s wr, 939 op/s 2024-04-23T18:23:17.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:17 smithi032 ceph-mon[29221]: pgmap v252: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.9 MiB/s wr, 939 op/s 2024-04-23T18:23:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 470.0 s, 2512.4 tps, lat 39.808 ms stddev 43.833 2024-04-23T18:23:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.078+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/495787406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc101120 msgr2=0x7ff6dc101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.078+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/495787406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc101120 0x7ff6dc101500 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009900 tx=0x7ff6cc02a410 comp rx=0 tx=0).stop 2024-04-23T18:23:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.082+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/495787406 shutdown_connections 2024-04-23T18:23:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.082+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/495787406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc1061a0 0x7ff6dc10dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.082+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/495787406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc101ad0 0x7ff6dc105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.082+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/495787406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc101120 0x7ff6dc101500 secure :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009900 tx=0x7ff6cc02a410 comp rx=0 tx=0).stop 2024-04-23T18:23:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.082+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/495787406 >> 172.21.15.32:0/495787406 conn(0x7ff6dc077fa0 msgr2=0x7ff6dc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.090+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/495787406 shutdown_connections 2024-04-23T18:23:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.093+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/495787406 wait complete. 2024-04-23T18:23:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.094+0000 7ff6e1dd8640 1 Processor -- start 2024-04-23T18:23:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.095+0000 7ff6e1dd8640 1 -- start start 2024-04-23T18:23:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc1061a0 0x7ff6dc197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 0x7ff6dc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 0x7ff6dc19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc111110 con 0x7ff6dc1061a0 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc110f90 con 0x7ff6dc197a90 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6e1dd8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc111290 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.096+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 0x7ff6dc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.097+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 0x7ff6dc197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:47460/0 (socket says 172.21.15.32:47460) 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.097+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/4021900030 learned_addr learned my addr 172.21.15.32:0/4021900030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.097+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 0x7ff6dc19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.098+0000 7ff6dbfff640 1 -- 172.21.15.32:0/4021900030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 msgr2=0x7ff6dc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.099+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 0x7ff6dc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.099+0000 7ff6dbfff640 1 -- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc1061a0 msgr2=0x7ff6dc197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.099+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc1061a0 0x7ff6dc197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.099+0000 7ff6dbfff640 1 -- 172.21.15.32:0/4021900030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc19d1d0 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.099+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 0x7ff6dc19ca00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009fd0 tx=0x7ff6d400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.103+0000 7ff6d97fa640 1 -- 172.21.15.32:0/4021900030 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d400ce20 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.103+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc19d4c0 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.103+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc19da10 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.103+0000 7ff6d97fa640 1 -- 172.21.15.32:0/4021900030 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d4005020 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.103+0000 7ff6d97fa640 1 -- 172.21.15.32:0/4021900030 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4010690 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.108+0000 7ff6e4bc8740 1 Processor -- start 2024-04-23T18:23:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 -- start start 2024-04-23T18:23:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6d97fa640 1 -- 172.21.15.32:0/4021900030 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff6d4010830 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f2ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f39800 0x561448f3bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561448e93970 con 0x561448f39800 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561448e304a0 con 0x561448f2e620 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561448e8f930 con 0x561448f2ef40 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.110+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f2ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.111+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f2ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:47476/0 (socket says 172.21.15.32:47476) 2024-04-23T18:23:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.111+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/3166227572 learned_addr learned my addr 172.21.15.32:0/3166227572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.111+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.113+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f39800 0x561448f3bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.116+0000 7ff6d97fa640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 0x7ff6c80a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6d97fa640 1 -- 172.21.15.32:0/4021900030 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7ff6d40d12c0 con 0x7ff6dc19c5a0 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6db7fe640 1 -- 172.21.15.32:0/3166227572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 msgr2=0x561448f2ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f2ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6db7fe640 1 -- 172.21.15.32:0/3166227572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f39800 msgr2=0x561448f3bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f39800 0x561448f3bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.117+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 0x7ff6c80a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.118+0000 7ff6db7fe640 1 -- 172.21.15.32:0/3166227572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561448ba6210 con 0x561448f2ef40 2024-04-23T18:23:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.118+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f392c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc1022f0 tx=0x7ff6cc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e1d26b9ff3fb67d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.118+0000 7ff6d1ffb640 1 -- 172.21.15.32:0/3166227572 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc03e070 con 0x561448f2ef40 2024-04-23T18:23:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.118+0000 7ff6d1ffb640 1 -- 172.21.15.32:0/3166227572 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6cc032da0 con 0x561448f2ef40 2024-04-23T18:23:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.118+0000 7ff6d1ffb640 1 -- 172.21.15.32:0/3166227572 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc035040 con 0x561448f2ef40 2024-04-23T18:23:19.113 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.119+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/3166227572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 msgr2=0x561448f392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.119+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f392c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc1022f0 tx=0x7ff6cc000bc0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.119+0000 7ff6dbfff640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 0x7ff6c80a63a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6d409d1b0 tx=0x7ff6d400c890 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/3166227572 shutdown_connections 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f39800 0x561448f3bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f392c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/3166227572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f2ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/3166227572 >> 172.21.15.32:0/3166227572 conn(0x561448f2ab30 msgr2=0x561448f2af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.120+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/3166227572 shutdown_connections 2024-04-23T18:23:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.121+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/3166227572 wait complete. 2024-04-23T18:23:19.115 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.121+0000 7ff6e4bc8740 1 -- start start 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.121+0000 7ff6e4bc8740 1 Processor -- start 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2e620 0x561448f390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f39800 0x561448f368e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561448f3c3d0 con 0x561448f2ef40 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561448e2e0b0 con 0x561448f39800 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561448f3c550 con 0x561448f2e620 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35774/0 (socket says 172.21.15.32:35774) 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2373792469 learned_addr learned my addr 172.21.15.32:0/2373792469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6e4bc8740 10 client.0 _ll_register_callbacks cb 0x561448e1bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.122+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2e620 0x561448f390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2373792469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2e620 msgr2=0x561448f390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2e620 0x561448f390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f39800 msgr2=0x561448f368e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f39800 0x561448f368e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2373792469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561448c72620 con 0x561448f2ef40 2024-04-23T18:23:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006010 tx=0x7ff6cc031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2344ea3975c7c899 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6baffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.123+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc04b070 con 0x561448f2ef40 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.124+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7ff6cc005340 con 0x561448f2ef40 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.124+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6cc032da0 con 0x561448f2ef40 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.124+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff6cc054050 con 0x561448f2ef40 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.124+0000 7ff6e4bc8740 10 client.16500 fetch_fsmap learned FSMap version 38 2024-04-23T18:23:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.124+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561448f482c0 con 0x561448f2ef40 2024-04-23T18:23:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6cc032da0 con 0x561448f2ef40 2024-04-23T18:23:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7ff6cc043450 con 0x561448f2ef40 2024-04-23T18:23:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 10 client.16500 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 20 client.16500 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f49610 0x561448f4ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 4 client.16500 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561448ea1390 con 0x561448f49610 2024-04-23T18:23:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f49610 0x561448f4ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6baffd640 10 client.16500 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.126+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f49610 0x561448f4ba00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57a8ef21c934e658 server_cookie=9b5d9e7bd12baec1 in_seq=0 out_seq=0 2024-04-23T18:23:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.127+0000 7ff6baffd640 1 -- 172.21.15.32:0/2373792469 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561448ea1390 con 0x561448f49610 2024-04-23T18:23:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.127+0000 7ff6baffd640 10 client.16500 handle_command_reply: tid=0 2024-04-23T18:23:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.127+0000 7ff6e4bc8740 1 client.16500 shutdown 2024-04-23T18:23:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 msgr2=0x561448f363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006010 tx=0x7ff6cc031750 comp rx=0 tx=0).stop 2024-04-23T18:23:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 shutdown_connections 2024-04-23T18:23:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f49610 0x561448f4ba00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f39800 0x561448f368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.128+0000 7ff6baffd640 0 client.16500 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ef40 0x561448f363a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2373792469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2e620 0x561448f390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 >> 172.21.15.32:0/2373792469 conn(0x561448f2ab30 msgr2=0x561448f308b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 shutdown_connections 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2373792469 wait complete. 2024-04-23T18:23:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.129+0000 7ff6e4bc8740 20 client.16500 trim_cache size 0 max 16384 2024-04-23T18:23:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 Processor -- start 2024-04-23T18:23:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 -- start start 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 0x561448f452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f31970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f38be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc011050 con 0x561448f2ab30 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00e8c0 con 0x561448f2e620 2024-04-23T18:23:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.148+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00c730 con 0x561448f2ef40 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f31970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f38be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 0x561448f452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 0x561448f452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35780/0 (socket says 172.21.15.32:35780) 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/1508261361 learned_addr learned my addr 172.21.15.32:0/1508261361 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/1508261361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 msgr2=0x561448f31970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f31970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/1508261361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 msgr2=0x561448f452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 0x561448f452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.149+0000 7ff6d2ffd640 1 -- 172.21.15.32:0/1508261361 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561448f2a960 con 0x561448f2ef40 2024-04-23T18:23:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.150+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f38be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x561448f2bf50 tx=0x7ff6c4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d018274c83204a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.150+0000 7ff6d17fa640 1 -- 172.21.15.32:0/1508261361 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4027070 con 0x561448f2ef40 2024-04-23T18:23:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.150+0000 7ff6d17fa640 1 -- 172.21.15.32:0/1508261361 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6c4021900 con 0x561448f2ef40 2024-04-23T18:23:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.150+0000 7ff6d17fa640 1 -- 172.21.15.32:0/1508261361 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4011a90 con 0x561448f2ef40 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/1508261361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 msgr2=0x561448f38be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f38be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x561448f2bf50 tx=0x7ff6c4004730 comp rx=0 tx=0).stop 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/1508261361 shutdown_connections 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f38be0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2e620 0x561448f31970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/1508261361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2ab30 0x561448f452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.151+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/1508261361 >> 172.21.15.32:0/1508261361 conn(0x561448f36a40 msgr2=0x561448f31260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.152+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/1508261361 shutdown_connections 2024-04-23T18:23:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.152+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/1508261361 wait complete. 2024-04-23T18:23:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.152+0000 7ff6e4bc8740 1 -- start start 2024-04-23T18:23:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 Processor -- start 2024-04-23T18:23:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 0x561448f3d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f3df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00cd80 con 0x561448f2e620 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00e1f0 con 0x561448f2ab30 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00d450 con 0x561448f2ef40 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35794/0 (socket says 172.21.15.32:35794) 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2788817812 learned_addr learned my addr 172.21.15.32:0/2788817812 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6e4bc8740 10 client.0 _ll_register_callbacks cb 0x7ff6cc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.153+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 0x561448f3d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f3df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2788817812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 msgr2=0x561448f3df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f3df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 msgr2=0x561448f3d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 0x561448f3d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 -- 172.21.15.32:0/2788817812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561448faec50 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 0x561448f3d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6db7fe640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc043cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a864c56f3e2b9901 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6baffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc04b070 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7ff6cc0046c0 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6c4021900 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.154+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff6cc043e60 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 10 client.16506 fetch_fsmap learned FSMap version 38 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561448faff40 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6c4021900 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7ff6cc004b30 con 0x561448f2e620 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 10 client.16506 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T18:23:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 20 client.16506 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f39800 0x561448fb25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 4 client.16506 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561448e9b200 con 0x561448f39800 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.155+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f39800 0x561448fb25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.156+0000 7ff6d2ffd640 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f39800 0x561448fb25c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e336bf97b1e1a37e server_cookie=2b62ac9888d89d62 in_seq=0 out_seq=0 2024-04-23T18:23:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.156+0000 7ff6baffd640 10 client.16506 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.167+0000 7ff6baffd640 1 -- 172.21.15.32:0/2788817812 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561448e9b200 con 0x561448f39800 2024-04-23T18:23:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.167+0000 7ff6baffd640 10 client.16506 handle_command_reply: tid=0 2024-04-23T18:23:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.167+0000 7ff6e4bc8740 1 client.16506 shutdown 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.168+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 msgr2=0x561448f3d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.168+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc043cc0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 shutdown_connections 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561448f39800 0x561448fb25c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561448f2ef40 0x561448f3df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561448f2e620 0x561448f3d9f0 secure :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc043cc0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 --2- 172.21.15.32:0/2788817812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561448f2ab30 0x561448f3d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 >> 172.21.15.32:0/2788817812 conn(0x561448f36a40 msgr2=0x561448f29cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6baffd640 0 client.16506 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 shutdown_connections 2024-04-23T18:23:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 1 -- 172.21.15.32:0/2788817812 wait complete. 2024-04-23T18:23:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.169+0000 7ff6e4bc8740 20 client.16506 trim_cache size 0 max 16384 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 msgr2=0x7ff6c80a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 0x7ff6c80a63a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6d409d1b0 tx=0x7ff6d400c890 comp rx=0 tx=0).stop 2024-04-23T18:23:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 msgr2=0x7ff6dc19ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 0x7ff6dc19ca00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009fd0 tx=0x7ff6d400ef10 comp rx=0 tx=0).stop 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 shutdown_connections 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6dc19c5a0 0x7ff6dc19ca00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6dc197a90 0x7ff6dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff6c80a3ee0 0x7ff6c80a63a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 --2- 172.21.15.32:0/4021900030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc1061a0 0x7ff6dc197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 >> 172.21.15.32:0/4021900030 conn(0x7ff6dc077fa0 msgr2=0x7ff6dc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 shutdown_connections 2024-04-23T18:23:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:19.170+0000 7ff6e1dd8640 1 -- 172.21.15.32:0/4021900030 wait complete. 2024-04-23T18:23:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:19 smithi040 ceph-mon[32626]: pgmap v253: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 886 op/s 2024-04-23T18:23:19.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:19 smithi064 ceph-mon[33062]: pgmap v253: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 886 op/s 2024-04-23T18:23:19.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:19 smithi032 ceph-mon[29221]: pgmap v253: 97 pgs: 97 active+clean; 3.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 886 op/s 2024-04-23T18:23:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:21 smithi040 ceph-mon[32626]: pgmap v254: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 887 op/s 2024-04-23T18:23:21.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:21 smithi064 ceph-mon[33062]: pgmap v254: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 887 op/s 2024-04-23T18:23:21.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:21 smithi032 ceph-mon[29221]: pgmap v254: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 887 op/s 2024-04-23T18:23:22.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:22 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:22.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:22 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:22.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:22 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:23 smithi064 ceph-mon[33062]: pgmap v255: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-23T18:23:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:23.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:23:23.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:23 smithi032 ceph-mon[29221]: pgmap v255: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-23T18:23:23.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:23.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:23.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:23:23.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:23 smithi040 ceph-mon[32626]: pgmap v255: 97 pgs: 97 active+clean; 3.9 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-23T18:23:23.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:23.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:23.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:23:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 475.0 s, 2105.4 tps, lat 47.287 ms stddev 90.770 2024-04-23T18:23:24.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:24 smithi064 ceph-mon[33062]: pgmap v256: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-23T18:23:24.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:24 smithi032 ceph-mon[29221]: pgmap v256: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-23T18:23:24.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:24 smithi040 ceph-mon[32626]: pgmap v256: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-23T18:23:25.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:25 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:25.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:25 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:25.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:25 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:26.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:26 smithi064 ceph-mon[33062]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-23T18:23:27.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:26 smithi032 ceph-mon[29221]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-23T18:23:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:26 smithi040 ceph-mon[32626]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-23T18:23:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 480.0 s, 2279.8 tps, lat 43.851 ms stddev 46.199 2024-04-23T18:23:28.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:28 smithi064 ceph-mon[33062]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:29.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:28 smithi032 ceph-mon[29221]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:28 smithi040 ceph-mon[32626]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:30.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:30 smithi064 ceph-mon[33062]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.96k op/s 2024-04-23T18:23:31.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:30 smithi032 ceph-mon[29221]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.96k op/s 2024-04-23T18:23:31.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:30 smithi040 ceph-mon[32626]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.96k op/s 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.302+0000 7f3aadae8640 1 -- 172.21.15.32:0/2556492022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa811b730 msgr2=0x7f3aa811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.302+0000 7f3aadae8640 1 --2- 172.21.15.32:0/2556492022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa811b730 0x7f3aa811db20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009860 tx=0x7f3a9c02a360 comp rx=0 tx=0).stop 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 -- 172.21.15.32:0/2556492022 shutdown_connections 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 --2- 172.21.15.32:0/2556492022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa811b730 0x7f3aa811db20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 --2- 172.21.15.32:0/2556492022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3aa810e7d0 0x7f3aa810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 --2- 172.21.15.32:0/2556492022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa81106f0 0x7f3aa810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 -- 172.21.15.32:0/2556492022 >> 172.21.15.32:0/2556492022 conn(0x7f3aa806d030 msgr2=0x7f3aa806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 -- 172.21.15.32:0/2556492022 shutdown_connections 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.303+0000 7f3aadae8640 1 -- 172.21.15.32:0/2556492022 wait complete. 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.304+0000 7f3aadae8640 1 Processor -- start 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.304+0000 7f3aadae8640 1 -- start start 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa810e7d0 0x7f3aa8118d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3aa81106f0 0x7f3aa8111de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8121020 con 0x7f3aa810e7d0 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8120ea0 con 0x7f3aa811b730 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aadae8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81211a0 con 0x7f3aa81106f0 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46256/0 (socket says 172.21.15.32:46256) 2024-04-23T18:23:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.305+0000 7f3aa7fff640 1 -- 172.21.15.32:0/1418806863 learned_addr learned my addr 172.21.15.32:0/1418806863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 -- 172.21.15.32:0/1418806863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3aa81106f0 msgr2=0x7f3aa8111de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:23:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3aa81106f0 0x7f3aa8111de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 -- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa810e7d0 msgr2=0x7f3aa8118d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:23:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa810e7d0 0x7f3aa8118d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 -- 172.21.15.32:0/1418806863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa8112910 con 0x7f3aa811b730 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa7fff640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02a950 tx=0x7f3a9c032cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa4ff9640 1 -- 172.21.15.32:0/1418806863 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c040020 con 0x7f3aa811b730 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.306+0000 7f3aa4ff9640 1 -- 172.21.15.32:0/1418806863 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9c038040 con 0x7f3aa811b730 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.307+0000 7f3aa4ff9640 1 -- 172.21.15.32:0/1418806863 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c037070 con 0x7f3aa811b730 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.307+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa8112ba0 con 0x7f3aa811b730 2024-04-23T18:23:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.307+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa8116ad0 con 0x7f3aa811b730 2024-04-23T18:23:32.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.310+0000 7f3ab0803740 1 Processor -- start 2024-04-23T18:23:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3aa4ff9640 1 -- 172.21.15.32:0/1418806863 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3a9c043020 con 0x7f3aa811b730 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 -- start start 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 0x5576a5cd0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5576a5c28970 con 0x5576a5cc3620 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5576a5bc54a0 con 0x5576a5cce800 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.311+0000 7f3ab0803740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5576a5c24930 con 0x5576a5cc3f40 2024-04-23T18:23:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36698/0 (socket says 172.21.15.32:36698) 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:47982/0 (socket says 172.21.15.32:47982) 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/4141102138 learned_addr learned my addr 172.21.15.32:0/4141102138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa4ff9640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 0x7f3a900a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 0x5576a5cd0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa4ff9640 1 -- 172.21.15.32:0/1418806863 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7f3a9c0f0100 con 0x7f3aa811b730 2024-04-23T18:23:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa7fff640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 0x7f3a900a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 -- 172.21.15.32:0/4141102138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 msgr2=0x5576a5cd0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 0x5576a5cd0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 -- 172.21.15.32:0/4141102138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 msgr2=0x5576a5cc3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa77fe640 1 -- 172.21.15.32:0/4141102138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a593b210 con 0x5576a5cc3f40 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.312+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 0x5576a5cd0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.313+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009a00 tx=0x7f3a98009ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420e5d2571d2260c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.313+0000 7f3aa7fff640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 0x7f3a900a62a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0eda30 tx=0x7f3a9c030a90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.313+0000 7f3a8dffb640 1 -- 172.21.15.32:0/4141102138 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800ec50 con 0x5576a5cc3f40 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.313+0000 7f3a8dffb640 1 -- 172.21.15.32:0/4141102138 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9800edf0 con 0x5576a5cc3f40 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.313+0000 7f3a8dffb640 1 -- 172.21.15.32:0/4141102138 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a980049a0 con 0x5576a5cc3f40 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- 172.21.15.32:0/4141102138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 msgr2=0x5576a5cce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009a00 tx=0x7f3a98009ec0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- 172.21.15.32:0/4141102138 shutdown_connections 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cce800 0x5576a5cd0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cce2c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 --2- 172.21.15.32:0/4141102138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- 172.21.15.32:0/4141102138 >> 172.21.15.32:0/4141102138 conn(0x5576a5cbfb30 msgr2=0x5576a5cbff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- 172.21.15.32:0/4141102138 shutdown_connections 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- 172.21.15.32:0/4141102138 wait complete. 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 -- start start 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 Processor -- start 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.314+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cc3620 0x5576a5cce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3f40 0x5576a5ccb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5576a5cd1430 con 0x5576a5cc3f40 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5576a5bc30b0 con 0x5576a5cc3620 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5576a5cd15b0 con 0x5576a5cce800 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:47984/0 (socket says 172.21.15.32:47984) 2024-04-23T18:23:32.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/1659615305 learned_addr learned my addr 172.21.15.32:0/1659615305 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cc3620 0x5576a5cce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3f40 0x5576a5ccb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3ab0803740 10 client.0 _ll_register_callbacks cb 0x5576a5bb0d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cc3620 msgr2=0x5576a5cce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cc3620 0x5576a5cce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/1659615305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3f40 msgr2=0x5576a5ccb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3f40 0x5576a5ccb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/1659615305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a5a07620 con 0x5576a5cce800 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0008f40 tx=0x7f3aa000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6b4d917db60d18a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.315+0000 7f3a86ffd640 10 client.26086 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.316+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9800edf0 con 0x5576a5cce800 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.316+0000 7f3a86ffd640 1 -- 172.21.15.32:0/1659615305 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0012030 con 0x5576a5cce800 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.316+0000 7f3a86ffd640 1 -- 172.21.15.32:0/1659615305 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7f3aa0003c10 con 0x5576a5cce800 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.316+0000 7f3a86ffd640 1 -- 172.21.15.32:0/1659615305 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3aa0003e50 con 0x5576a5cce800 2024-04-23T18:23:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 10 client.26086 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({fsmap=39}) -- 0x5576a5cdd1f0 con 0x5576a5cce800 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3a86ffd640 1 -- 172.21.15.32:0/1659615305 <== mon.1 v2:172.21.15.64:3300/0 4 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f3aa0004020 con 0x5576a5cce800 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 10 client.26086 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 10 client.26086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 20 client.26086 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cde5f0 0x5576a5ce0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 4 client.26086 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a9800edf0 con 0x5576a5cde5f0 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.317+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cde5f0 0x5576a5ce0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3a86ffd640 10 client.26086 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cde5f0 0x5576a5ce0a20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8db9a870f933c3 server_cookie=651e3336aef16138 in_seq=0 out_seq=0 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3a86ffd640 1 -- 172.21.15.32:0/1659615305 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3a9800edf0 con 0x5576a5cde5f0 2024-04-23T18:23:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3a86ffd640 10 client.26086 handle_command_reply: tid=0 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3ab0803740 1 client.26086 shutdown 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 msgr2=0x5576a5ccb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.318+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0008f40 tx=0x7f3aa000b040 comp rx=0 tx=0).stop 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 shutdown_connections 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cde5f0 0x5576a5ce0a20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3a86ffd640 0 client.26086 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cce800 0x5576a5ccb8e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3f40 0x5576a5ccb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 --2- 172.21.15.32:0/1659615305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cc3620 0x5576a5cce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 >> 172.21.15.32:0/1659615305 conn(0x5576a5cbfb30 msgr2=0x5576a5cc58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 shutdown_connections 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 1 -- 172.21.15.32:0/1659615305 wait complete. 2024-04-23T18:23:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.319+0000 7f3ab0803740 20 client.26086 trim_cache size 0 max 16384 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 Processor -- start 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 -- start start 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cda2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5ccdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c7a50 con 0x5576a5cc3620 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c52f0 con 0x5576a5cbfb30 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.340+0000 7f3ab0803740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c3160 con 0x5576a5cc3f40 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5ccdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36716/0 (socket says 172.21.15.32:36716) 2024-04-23T18:23:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa77fe640 1 -- 172.21.15.32:0/3695178144 learned_addr learned my addr 172.21.15.32:0/3695178144 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cda2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/3695178144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 msgr2=0x5576a5cda2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cda2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/3695178144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 msgr2=0x5576a5cc6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/3695178144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a5cc08c0 con 0x5576a5cc3f40 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.341+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5ccdbe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000f80 tx=0x7f3aa0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccd4ce1b9aff090b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.342+0000 7f3a8d7fa640 1 -- 172.21.15.32:0/3695178144 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00095f0 con 0x5576a5cc3f40 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.342+0000 7f3a8d7fa640 1 -- 172.21.15.32:0/3695178144 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa0010070 con 0x5576a5cc3f40 2024-04-23T18:23:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.342+0000 7f3a8d7fa640 1 -- 172.21.15.32:0/3695178144 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0015780 con 0x5576a5cc3f40 2024-04-23T18:23:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- 172.21.15.32:0/3695178144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 msgr2=0x5576a5ccdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5ccdbe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000f80 tx=0x7f3aa0014000 comp rx=0 tx=0).stop 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- 172.21.15.32:0/3695178144 shutdown_connections 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5ccdbe0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cc6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3695178144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cda2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- 172.21.15.32:0/3695178144 >> 172.21.15.32:0/3695178144 conn(0x5576a5ccba40 msgr2=0x5576a5cc6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- 172.21.15.32:0/3695178144 shutdown_connections 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- 172.21.15.32:0/3695178144 wait complete. 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.343+0000 7f3ab0803740 1 -- start start 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 Processor -- start 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cd6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cd6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cc76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c37b0 con 0x5576a5cc3620 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c4c20 con 0x5576a5cbfb30 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81c3e80 con 0x5576a5cc3f40 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3ab0803740 10 client.0 _ll_register_callbacks cb 0x7f3aa0008d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cd6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cd6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3aa6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cd6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:46292/0 (socket says 172.21.15.32:46292) 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3aa6ffd640 1 -- 172.21.15.32:0/3192643285 learned_addr learned my addr 172.21.15.32:0/3192643285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.344+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cc76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 -- 172.21.15.32:0/3192643285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 msgr2=0x5576a5cc76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cc76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 -- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 msgr2=0x5576a5cd6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cd6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 -- 172.21.15.32:0/3192643285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a5cc7de0 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3aa77fe640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cd6d40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009840 tx=0x7f3a9800eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcd97053db6dcdb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3a86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98005310 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7f3a980054b0 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa0010070 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3a98038050 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3ab0803740 10 client.16521 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5576a5d44590 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.345+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa0010070 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f3a980035f0 con 0x5576a5cc3620 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 10 client.16521 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 20 client.16521 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cce800 0x5576a5d46d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 4 client.16521 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576a5c36390 con 0x5576a5cce800 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cce800 0x5576a5d46d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3a86ffd640 10 client.16521 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.346+0000 7f3aa6ffd640 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cce800 0x5576a5d46d20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76964ef4b1dbf3b5 server_cookie=57bc13ad9a70d93e in_seq=0 out_seq=0 2024-04-23T18:23:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.359+0000 7f3a86ffd640 1 -- 172.21.15.32:0/3192643285 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576a5c36390 con 0x5576a5cce800 2024-04-23T18:23:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.359+0000 7f3a86ffd640 10 client.16521 handle_command_reply: tid=0 2024-04-23T18:23:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.359+0000 7f3ab0803740 1 client.16521 shutdown 2024-04-23T18:23:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 msgr2=0x5576a5cd6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cd6d40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009840 tx=0x7f3a9800eb60 comp rx=0 tx=0).stop 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 shutdown_connections 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5576a5cce800 0x5576a5d46d20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576a5cc3f40 0x5576a5cc76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a5cc3620 0x5576a5cd6d40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 --2- 172.21.15.32:0/3192643285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5576a5cbfb30 0x5576a5cd6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3a86ffd640 0 client.16521 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 >> 172.21.15.32:0/3192643285 conn(0x5576a5ccba40 msgr2=0x5576a5cc4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 shutdown_connections 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 1 -- 172.21.15.32:0/3192643285 wait complete. 2024-04-23T18:23:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.360+0000 7f3ab0803740 20 client.16521 trim_cache size 0 max 16384 2024-04-23T18:23:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:23:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:23:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:23:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:23:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 msgr2=0x7f3a900a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 0x7f3a900a62a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0eda30 tx=0x7f3a9c030a90 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 msgr2=0x7f3aa8112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02a950 tx=0x7f3a9c032cd0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 shutdown_connections 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa811b730 0x7f3aa8112320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3aa81106f0 0x7f3aa8111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3a900a3de0 0x7f3a900a62a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 --2- 172.21.15.32:0/1418806863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa810e7d0 0x7f3aa8118d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 >> 172.21.15.32:0/1418806863 conn(0x7f3aa806d030 msgr2=0x7f3aa811a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 shutdown_connections 2024-04-23T18:23:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:32.361+0000 7f3aadae8640 1 -- 172.21.15.32:0/1418806863 wait complete. 2024-04-23T18:23:32.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:32 smithi064 ceph-mon[33062]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:33.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:32 smithi032 ceph-mon[29221]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:32 smithi040 ceph-mon[32626]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-23T18:23:33.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 485.0 s, 2817.0 tps, lat 35.550 ms stddev 38.440 2024-04-23T18:23:34.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:34 smithi064 ceph-mon[33062]: pgmap v261: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-04-23T18:23:35.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:34 smithi032 ceph-mon[29221]: pgmap v261: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-04-23T18:23:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:34 smithi040 ceph-mon[32626]: pgmap v261: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-04-23T18:23:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:36 smithi032 ceph-mon[29221]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-23T18:23:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:36 smithi040 ceph-mon[32626]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-23T18:23:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:36 smithi064 ceph-mon[33062]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-23T18:23:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 490.0 s, 2720.6 tps, lat 36.855 ms stddev 39.120 2024-04-23T18:23:39.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:38 smithi032 ceph-mon[29221]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-23T18:23:39.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:38 smithi040 ceph-mon[32626]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-23T18:23:39.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:38 smithi064 ceph-mon[33062]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-23T18:23:40.973 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:40 smithi064 ceph-mon[33062]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.1 MiB/s wr, 850 op/s 2024-04-23T18:23:41.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:40 smithi032 ceph-mon[29221]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.1 MiB/s wr, 850 op/s 2024-04-23T18:23:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:40 smithi040 ceph-mon[32626]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.1 MiB/s wr, 850 op/s 2024-04-23T18:23:43.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:42 smithi032 ceph-mon[29221]: pgmap v265: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-23T18:23:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:42 smithi040 ceph-mon[32626]: pgmap v265: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-23T18:23:43.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:42 smithi064 ceph-mon[33062]: pgmap v265: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-23T18:23:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 495.0 s, 2427.4 tps, lat 41.092 ms stddev 44.928 2024-04-23T18:23:45.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:44 smithi032 ceph-mon[29221]: pgmap v266: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-04-23T18:23:45.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:44 smithi040 ceph-mon[32626]: pgmap v266: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-04-23T18:23:45.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:44 smithi064 ceph-mon[33062]: pgmap v266: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-04-23T18:23:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.477+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2612372147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c810e660 msgr2=0x7fe1c810ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.477+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2612372147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c810e660 0x7fe1c810ea40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009900 tx=0x7fe1bc02a410 comp rx=0 tx=0).stop 2024-04-23T18:23:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2612372147 shutdown_connections 2024-04-23T18:23:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2612372147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c8108f60 0x7fe1c8115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2612372147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c8108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2612372147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c810e660 0x7fe1c810ea40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2612372147 >> 172.21.15.32:0/2612372147 conn(0x7fe1c8076490 msgr2=0x7fe1c80788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2612372147 shutdown_connections 2024-04-23T18:23:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2612372147 wait complete. 2024-04-23T18:23:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.478+0000 7fe1cd6ae640 1 Processor -- start 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 -- start start 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c8108f60 0x7fe1c819f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c8118850 con 0x7fe1c8108f60 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81186d0 con 0x7fe1c810e660 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1cd6ae640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81189d0 con 0x7fe1c81085a0 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c8108f60 0x7fe1c819f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49072/0 (socket says 172.21.15.32:49072) 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 -- 172.21.15.32:0/2124393095 learned_addr learned my addr 172.21.15.32:0/2124393095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 -- 172.21.15.32:0/2124393095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 msgr2=0x7fe1c819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 -- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c8108f60 msgr2=0x7fe1c819f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c8108f60 0x7fe1c819f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 -- 172.21.15.32:0/2124393095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c81a3d20 con 0x7fe1c810e660 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.479+0000 7fe1c77fe640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000eb10 tx=0x7fe1c000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.480+0000 7fe1affff640 1 -- 172.21.15.32:0/2124393095 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0004b20 con 0x7fe1c810e660 2024-04-23T18:23:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.480+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c81a4010 con 0x7fe1c810e660 2024-04-23T18:23:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.480+0000 7fe1affff640 1 -- 172.21.15.32:0/2124393095 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1c0004cc0 con 0x7fe1c810e660 2024-04-23T18:23:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.480+0000 7fe1affff640 1 -- 172.21.15.32:0/2124393095 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0010670 con 0x7fe1c810e660 2024-04-23T18:23:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.480+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c81ab8a0 con 0x7fe1c810e660 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.483+0000 7fe1affff640 1 -- 172.21.15.32:0/2124393095 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe1c0005020 con 0x7fe1c810e660 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.483+0000 7fe1d03c6740 1 Processor -- start 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.483+0000 7fe1d03c6740 1 -- start start 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 0x561c4ad402c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad42bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561c4ac9a970 con 0x561c4ad35f40 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561c4ac374a0 con 0x561c4ad35620 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561c4ac96930 con 0x561c4ad40800 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1affff640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 0x7fe1b00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 0x561c4ad402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 0x561c4ad402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50108/0 (socket says 172.21.15.32:50108) 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c6ffd640 1 -- 172.21.15.32:0/4194426052 learned_addr learned my addr 172.21.15.32:0/4194426052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c77fe640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 0x7fe1b00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad42bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1affff640 1 -- 172.21.15.32:0/2124393095 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fe1c00d0d50 con 0x7fe1c810e660 2024-04-23T18:23:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 -- 172.21.15.32:0/4194426052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 msgr2=0x561c4ad42bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad42bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 -- 172.21.15.32:0/4194426052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 msgr2=0x561c4ad402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 0x561c4ad402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 -- 172.21.15.32:0/4194426052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c4a9ad210 con 0x561c4ad35620 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad35a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009710 tx=0x7fe1b80096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290551c0945fd085 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.484+0000 7fe1c77fe640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 0x7fe1b00a61d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00ce570 tx=0x7fe1c001bb30 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.485+0000 7fe1ad7fa640 1 -- 172.21.15.32:0/4194426052 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8004030 con 0x561c4ad35620 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.485+0000 7fe1ad7fa640 1 -- 172.21.15.32:0/4194426052 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x561c4ad35620 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.485+0000 7fe1ad7fa640 1 -- 172.21.15.32:0/4194426052 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80107e0 con 0x561c4ad35620 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- 172.21.15.32:0/4194426052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 msgr2=0x561c4ad35a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad35a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009710 tx=0x7fe1b80096a0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- 172.21.15.32:0/4194426052 shutdown_connections 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad42bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35f40 0x561c4ad402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/4194426052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad35a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- 172.21.15.32:0/4194426052 >> 172.21.15.32:0/4194426052 conn(0x561c4ad31b30 msgr2=0x561c4ad31f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- 172.21.15.32:0/4194426052 shutdown_connections 2024-04-23T18:23:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- 172.21.15.32:0/4194426052 wait complete. 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- start start 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 Processor -- start 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35f40 0x561c4ad3d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561c4ad43430 con 0x561c4ad40800 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561c4ac350b0 con 0x561c4ad35620 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561c4ad435b0 con 0x561c4ad35f40 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.486+0000 7fe1d03c6740 10 client.0 _ll_register_callbacks cb 0x561c4ac22d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50114/0 (socket says 172.21.15.32:50114) 2024-04-23T18:23:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 -- 172.21.15.32:0/382005628 learned_addr learned my addr 172.21.15.32:0/382005628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35f40 0x561c4ad3d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 -- 172.21.15.32:0/382005628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35f40 msgr2=0x561c4ad3d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35f40 0x561c4ad3d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 -- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 msgr2=0x561c4ad401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 -- 172.21.15.32:0/382005628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c4aa79620 con 0x561c4ad40800 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80133f0 tx=0x7fe1b8003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf0ee4c11988594 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.487+0000 7fe1aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80039b0 con 0x561c4ad40800 2024-04-23T18:23:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b80035f0 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fe1b802b030 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe1b80039b0 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1d03c6740 10 client.16533 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.488+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561c4ad4f3e0 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.489+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.489+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7fe1b801e070 con 0x561c4ad40800 2024-04-23T18:23:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.489+0000 7fe1d03c6740 10 client.16533 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.490+0000 7fe1d03c6740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.490+0000 7fe1d03c6740 20 client.16533 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.490+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4ad52b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.490+0000 7fe1d03c6740 4 client.16533 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.490+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c4aca8390 con 0x561c4ad50730 2024-04-23T18:23:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.492+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4ad52b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.493+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4ad52b20 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663bdf06b0707c7d server_cookie=e0660dde169c547c in_seq=0 out_seq=0 2024-04-23T18:23:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.493+0000 7fe1aaff5640 10 client.16533 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.503+0000 7fe1aaff5640 1 -- 172.21.15.32:0/382005628 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c4aca8390 con 0x561c4ad50730 2024-04-23T18:23:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.503+0000 7fe1aaff5640 10 client.16533 handle_command_reply: tid=0 2024-04-23T18:23:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.507+0000 7fe1d03c6740 1 client.16533 shutdown 2024-04-23T18:23:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.513+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 msgr2=0x561c4ad3dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.513+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80133f0 tx=0x7fe1b8003f00 comp rx=0 tx=0).stop 2024-04-23T18:23:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.518+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 shutdown_connections 2024-04-23T18:23:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.518+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4ad52b20 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.518+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad3dae0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.519+0000 7fe1aaff5640 0 client.16533 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.519+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35f40 0x561c4ad3d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.519+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/382005628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35620 0x561c4ad401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.519+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 >> 172.21.15.32:0/382005628 conn(0x561c4ad31b30 msgr2=0x561c4ad378b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.525+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 shutdown_connections 2024-04-23T18:23:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.528+0000 7fe1d03c6740 1 -- 172.21.15.32:0/382005628 wait complete. 2024-04-23T18:23:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.530+0000 7fe1d03c6740 20 client.16533 trim_cache size 0 max 16384 2024-04-23T18:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 Processor -- start 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 -- start start 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad3fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad58260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b6d00 con 0x561c4ad35620 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b45a0 con 0x561c4ad35f40 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.561+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b2390 con 0x561c4ad40800 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50126/0 (socket says 172.21.15.32:50126) 2024-04-23T18:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 -- 172.21.15.32:0/1723112581 learned_addr learned my addr 172.21.15.32:0/1723112581 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad3fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad58260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 -- 172.21.15.32:0/1723112581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 msgr2=0x561c4ad58260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad58260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 -- 172.21.15.32:0/1723112581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 msgr2=0x561c4ad3fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad3fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 -- 172.21.15.32:0/1723112581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c4ad35d30 con 0x561c4ad35620 2024-04-23T18:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.562+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800ea60 tx=0x7fe1b8013510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20533ac1ef4b79 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.564+0000 7fe1acff9640 1 -- 172.21.15.32:0/1723112581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8025040 con 0x561c4ad35620 2024-04-23T18:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.564+0000 7fe1acff9640 1 -- 172.21.15.32:0/1723112581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x561c4ad35620 2024-04-23T18:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.564+0000 7fe1acff9640 1 -- 172.21.15.32:0/1723112581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8035080 con 0x561c4ad35620 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.565+0000 7fe1d03c6740 1 -- 172.21.15.32:0/1723112581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 msgr2=0x561c4ad4c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.565+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800ea60 tx=0x7fe1b8013510 comp rx=0 tx=0).stop 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 -- 172.21.15.32:0/1723112581 shutdown_connections 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad40800 0x561c4ad58260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad3fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/1723112581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad35620 0x561c4ad4c2f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 -- 172.21.15.32:0/1723112581 >> 172.21.15.32:0/1723112581 conn(0x561c4ad31b30 msgr2=0x561c4ad30cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 -- 172.21.15.32:0/1723112581 shutdown_connections 2024-04-23T18:23:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 -- 172.21.15.32:0/1723112581 wait complete. 2024-04-23T18:23:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 -- start start 2024-04-23T18:23:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.566+0000 7fe1d03c6740 1 Processor -- start 2024-04-23T18:23:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad43f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad44450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b2a60 con 0x561c4ad40800 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b3ed0 con 0x561c4ad35f40 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c81b3130 con 0x561c4ad35620 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1d03c6740 10 client.0 _ll_register_callbacks cb 0x7fe1b80043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1c67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:42424/0 (socket says 172.21.15.32:42424) 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1c67fc640 1 -- 172.21.15.32:0/3328614395 learned_addr learned my addr 172.21.15.32:0/3328614395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad43f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.567+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad44450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 -- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 msgr2=0x561c4ad43f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad43f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 -- 172.21.15.32:0/3328614395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 msgr2=0x561c4ad44450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad44450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 -- 172.21.15.32:0/3328614395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c4ad44c80 con 0x561c4ad35620 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1c67fc640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8004330 tx=0x7fe1b800e870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82f2c9b21766b551 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8025070 con 0x561c4ad35620 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fe1b8003b50 con 0x561c4ad35620 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.568+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b80035f0 con 0x561c4ad35620 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe1b80039b0 con 0x561c4ad35620 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 10 client.26110 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561c4adb6ad0 con 0x561c4ad35620 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x561c4ad35620 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7fe1b801e070 con 0x561c4ad35620 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 10 client.26110 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 10 client.26110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 20 client.26110 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4adb8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 4 client.26110 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561c4aca2200 con 0x561c4ad50730 2024-04-23T18:23:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.569+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4adb8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.570+0000 7fe1c6ffd640 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4adb8ec0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409b7b9c3d33d7f6 server_cookie=6f05ab2d7e3d7dc7 in_seq=0 out_seq=0 2024-04-23T18:23:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.570+0000 7fe1aaff5640 10 client.26110 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1aaff5640 1 -- 172.21.15.32:0/3328614395 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561c4aca2200 con 0x561c4ad50730 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1aaff5640 10 client.26110 handle_command_reply: tid=0 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 client.26110 shutdown 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 msgr2=0x561c4ad48ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8004330 tx=0x7fe1b800e870 comp rx=0 tx=0).stop 2024-04-23T18:23:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 shutdown_connections 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561c4ad50730 0x561c4adb8ec0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561c4ad40800 0x561c4ad44450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c4ad35f40 0x561c4ad43f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1aaff5640 0 client.26110 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 --2- 172.21.15.32:0/3328614395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c4ad35620 0x561c4ad48ba0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.590+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 >> 172.21.15.32:0/3328614395 conn(0x561c4ad31b30 msgr2=0x561c4ad4b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 shutdown_connections 2024-04-23T18:23:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1d03c6740 1 -- 172.21.15.32:0/3328614395 wait complete. 2024-04-23T18:23:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1d03c6740 20 client.26110 trim_cache size 0 max 16384 2024-04-23T18:23:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:23:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:23:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:23:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 msgr2=0x7fe1b00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 0x7fe1b00a61d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00ce570 tx=0x7fe1c001bb30 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 msgr2=0x7fe1c81a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000eb10 tx=0x7fe1c000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 shutdown_connections 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.591+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1c810e660 0x7fe1c81a35f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe1b00a3d10 0x7fe1b00a61d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c8108f60 0x7fe1c819f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 --2- 172.21.15.32:0/2124393095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1c81085a0 0x7fe1c819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 >> 172.21.15.32:0/2124393095 conn(0x7fe1c8076490 msgr2=0x7fe1c8077d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 shutdown_connections 2024-04-23T18:23:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:45.592+0000 7fe1cd6ae640 1 -- 172.21.15.32:0/2124393095 wait complete. 2024-04-23T18:23:47.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:46 smithi032 ceph-mon[29221]: pgmap v267: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.5 MiB/s wr, 778 op/s 2024-04-23T18:23:47.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:46 smithi040 ceph-mon[32626]: pgmap v267: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.5 MiB/s wr, 778 op/s 2024-04-23T18:23:47.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:46 smithi064 ceph-mon[33062]: pgmap v267: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.5 MiB/s wr, 778 op/s 2024-04-23T18:23:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 500.0 s, 2764.0 tps, lat 36.272 ms stddev 40.209 2024-04-23T18:23:49.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:48 smithi032 ceph-mon[29221]: pgmap v268: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 631 op/s 2024-04-23T18:23:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:48 smithi040 ceph-mon[32626]: pgmap v268: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 631 op/s 2024-04-23T18:23:49.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:48 smithi064 ceph-mon[33062]: pgmap v268: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 631 op/s 2024-04-23T18:23:51.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:50 smithi032 ceph-mon[29221]: pgmap v269: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.3 MiB/s wr, 934 op/s 2024-04-23T18:23:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:50 smithi040 ceph-mon[32626]: pgmap v269: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.3 MiB/s wr, 934 op/s 2024-04-23T18:23:51.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:50 smithi064 ceph-mon[33062]: pgmap v269: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.3 MiB/s wr, 934 op/s 2024-04-23T18:23:53.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:52 smithi032 ceph-mon[29221]: pgmap v270: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 648 op/s 2024-04-23T18:23:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:52 smithi040 ceph-mon[32626]: pgmap v270: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 648 op/s 2024-04-23T18:23:53.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:52 smithi064 ceph-mon[33062]: pgmap v270: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 648 op/s 2024-04-23T18:23:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 505.0 s, 2473.2 tps, lat 40.274 ms stddev 45.789 2024-04-23T18:23:55.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:54 smithi032 ceph-mon[29221]: pgmap v271: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 699 op/s 2024-04-23T18:23:55.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:54 smithi040 ceph-mon[32626]: pgmap v271: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 699 op/s 2024-04-23T18:23:55.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:54 smithi064 ceph-mon[33062]: pgmap v271: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 699 op/s 2024-04-23T18:23:57.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:56 smithi032 ceph-mon[29221]: pgmap v272: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 773 op/s 2024-04-23T18:23:57.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:56 smithi040 ceph-mon[32626]: pgmap v272: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 773 op/s 2024-04-23T18:23:57.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:56 smithi064 ceph-mon[33062]: pgmap v272: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 773 op/s 2024-04-23T18:23:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 -- 172.21.15.32:0/1582061518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 msgr2=0x7fb8e010ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 --2- 172.21.15.32:0/1582061518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e010ee00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009c10 tx=0x7fb8d802a6f0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 -- 172.21.15.32:0/1582061518 shutdown_connections 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 --2- 172.21.15.32:0/1582061518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e010ee00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 --2- 172.21.15.32:0/1582061518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e006a140 0x7fb8e006a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 --2- 172.21.15.32:0/1582061518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0108080 0x7fb8e0108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.717+0000 7fb8e513d640 1 -- 172.21.15.32:0/1582061518 >> 172.21.15.32:0/1582061518 conn(0x7fb8e0077d30 msgr2=0x7fb8e0078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.718+0000 7fb8e513d640 1 -- 172.21.15.32:0/1582061518 shutdown_connections 2024-04-23T18:23:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.718+0000 7fb8e513d640 1 -- 172.21.15.32:0/1582061518 wait complete. 2024-04-23T18:23:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.719+0000 7fb8e513d640 1 Processor -- start 2024-04-23T18:23:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.719+0000 7fb8e513d640 1 -- start start 2024-04-23T18:23:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 0x7fb8e019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01121a0 con 0x7fb8e006ab00 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e0112020 con 0x7fb8e006a140 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8e513d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e0112320 con 0x7fb8e0108080 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:37604/0 (socket says 172.21.15.32:37604) 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8ded9d640 1 -- 172.21.15.32:0/352959147 learned_addr learned my addr 172.21.15.32:0/352959147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8df59e640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 0x7fb8e019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.720+0000 7fb8ded9d640 1 -- 172.21.15.32:0/352959147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 msgr2=0x7fb8e019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 0x7fb8e019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8ded9d640 1 -- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 msgr2=0x7fb8e0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e0198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8ded9d640 1 -- 172.21.15.32:0/352959147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e019d670 con 0x7fb8e006a140 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8df59e640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 0x7fb8e019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8de59c640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000d950 tx=0x7fb8d000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8c7fff640 1 -- 172.21.15.32:0/352959147 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0017070 con 0x7fb8e006a140 2024-04-23T18:23:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8c7fff640 1 -- 172.21.15.32:0/352959147 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d0009c30 con 0x7fb8e006a140 2024-04-23T18:23:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e019d960 con 0x7fb8e006a140 2024-04-23T18:23:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8c7fff640 1 -- 172.21.15.32:0/352959147 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000f620 con 0x7fb8e006a140 2024-04-23T18:23:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.721+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e01a51f0 con 0x7fb8e006a140 2024-04-23T18:23:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.724+0000 7fb8c7fff640 1 -- 172.21.15.32:0/352959147 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb8d0010460 con 0x7fb8e006a140 2024-04-23T18:23:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 Processor -- start 2024-04-23T18:23:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 -- start start 2024-04-23T18:23:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c1c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c2b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616e0b81970 con 0x5616e0c1c560 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5616e0b1e4a0 con 0x5616e0c1ce80 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5616e0b7d930 con 0x5616e0c2bc10 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8c7fff640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 0x7fb8c80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8df59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c1c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c2b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39466/0 (socket says 172.21.15.32:39466) 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8df59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c1c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42088/0 (socket says 172.21.15.32:42088) 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8de59c640 1 -- 172.21.15.32:0/2259189022 learned_addr learned my addr 172.21.15.32:0/2259189022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.725+0000 7fb8c7fff640 1 -- 172.21.15.32:0/352959147 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fb8d00d03d0 con 0x7fb8e006a140 2024-04-23T18:23:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 -- 172.21.15.32:0/2259189022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 msgr2=0x5616e0c2b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c2b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 -- 172.21.15.32:0/2259189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 msgr2=0x5616e0c1c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8df59e640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 0x7fb8c80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c1c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 -- 172.21.15.32:0/2259189022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e0894210 con 0x5616e0c2bc10 2024-04-23T18:23:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8de59c640 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40114f0 tx=0x7fb8d40119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d43d5568bc8445b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8df59e640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 0x7fb8c80a6250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004af0 tx=0x7fb8d802a6f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8c57fa640 1 -- 172.21.15.32:0/2259189022 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4005020 con 0x5616e0c2bc10 2024-04-23T18:23:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8c57fa640 1 -- 172.21.15.32:0/2259189022 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d40047a0 con 0x5616e0c2bc10 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.726+0000 7fb8c57fa640 1 -- 172.21.15.32:0/2259189022 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d400f920 con 0x5616e0c2bc10 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 -- 172.21.15.32:0/2259189022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 msgr2=0x5616e0c2e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40114f0 tx=0x7fb8d40119b0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 -- 172.21.15.32:0/2259189022 shutdown_connections 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c2e000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c2b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/2259189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c1c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 -- 172.21.15.32:0/2259189022 >> 172.21.15.32:0/2259189022 conn(0x5616e0c18b30 msgr2=0x5616e0c18f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 -- 172.21.15.32:0/2259189022 shutdown_connections 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.727+0000 7fb8e7e58740 1 -- 172.21.15.32:0/2259189022 wait complete. 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 -- start start 2024-04-23T18:23:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 Processor -- start 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c28730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c28c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616e0c2e880 con 0x5616e0c1c560 2024-04-23T18:23:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5616e0b1c0b0 con 0x5616e0c1ce80 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42094/0 (socket says 172.21.15.32:42094) 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5616e0c2ea00 con 0x5616e0c2bc10 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8ded9d640 1 -- 172.21.15.32:0/1194287525 learned_addr learned my addr 172.21.15.32:0/1194287525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8e7e58740 10 client.0 _ll_register_callbacks cb 0x5616e0b09d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8de59c640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c28730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.728+0000 7fb8df59e640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c28c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 -- 172.21.15.32:0/1194287525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 msgr2=0x5616e0c28c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c28c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 -- 172.21.15.32:0/1194287525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 msgr2=0x5616e0c2b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 -- 172.21.15.32:0/1194287525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e0960620 con 0x5616e0c1ce80 2024-04-23T18:23:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8de59c640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c28730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d4005130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536c1f07f1236c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4028070 con 0x5616e0c1ce80 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.729+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fb8d4004020 con 0x5616e0c1ce80 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.730+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d40047a0 con 0x5616e0c1ce80 2024-04-23T18:23:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.730+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb8d4031050 con 0x5616e0c1ce80 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.730+0000 7fb8e7e58740 10 client.26114 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.730+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5616e0c3a6f0 con 0x5616e0c1ce80 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d40047a0 con 0x5616e0c1ce80 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7fb8d400fb50 con 0x5616e0c1ce80 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 10 client.26114 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 20 client.26114 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c3bb50 0x5616e0c3df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 4 client.26114 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616e0b8f390 con 0x5616e0c3bb50 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.731+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c3bb50 0x5616e0c3df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.732+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c3bb50 0x5616e0c3df40 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d6b59d5702815f server_cookie=6d9660a520ee1759 in_seq=0 out_seq=0 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.732+0000 7fb8c2ff5640 10 client.26114 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.733+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/1194287525 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616e0b8f390 con 0x5616e0c3bb50 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.733+0000 7fb8c2ff5640 10 client.26114 handle_command_reply: tid=0 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.733+0000 7fb8e7e58740 1 client.26114 shutdown 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.733+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 msgr2=0x5616e0c28730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.733+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c28730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d4005130 comp rx=0 tx=0).stop 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 shutdown_connections 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c3bb50 0x5616e0c3df40 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8c2ff5640 0 client.26114 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c2bc10 0x5616e0c28c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c1ce80 0x5616e0c28730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/1194287525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1c560 0x5616e0c2b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 >> 172.21.15.32:0/1194287525 conn(0x5616e0c18b30 msgr2=0x5616e0c1e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 shutdown_connections 2024-04-23T18:23:58.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 1 -- 172.21.15.32:0/1194287525 wait complete. 2024-04-23T18:23:58.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.734+0000 7fb8e7e58740 20 client.26114 trim_cache size 0 max 16384 2024-04-23T18:23:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8e7e58740 1 Processor -- start 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8e7e58740 1 -- start start 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c2af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39492/0 (socket says 172.21.15.32:39492) 2024-04-23T18:23:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c2af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c2af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42106/0 (socket says 172.21.15.32:42106) 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.758+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8ded9d640 1 -- 172.21.15.32:0/3457283661 learned_addr learned my addr 172.21.15.32:0/3457283661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8df59e640 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01ae340 con 0x5616e0c1ce80 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01abbe0 con 0x5616e0c2bc10 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.759+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a9a50 con 0x5616e0c1c560 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 -- 172.21.15.32:0/3457283661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 msgr2=0x5616e0c436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 -- 172.21.15.32:0/3457283661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 msgr2=0x5616e0c2af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c2af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 -- 172.21.15.32:0/3457283661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e0c2e6c0 con 0x5616e0c1c560 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00a4a70 tx=0x7fb8d00c7440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6259e28aecea08a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8c4ff9640 1 -- 172.21.15.32:0/3457283661 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0020040 con 0x5616e0c1c560 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8c4ff9640 1 -- 172.21.15.32:0/3457283661 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d00a06e0 con 0x5616e0c1c560 2024-04-23T18:23:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.760+0000 7fb8c4ff9640 1 -- 172.21.15.32:0/3457283661 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00a0170 con 0x5616e0c1c560 2024-04-23T18:23:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.761+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 msgr2=0x5616e0c377b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.761+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00a4a70 tx=0x7fb8d00c7440 comp rx=0 tx=0).stop 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 shutdown_connections 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c2af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/3457283661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c377b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 >> 172.21.15.32:0/3457283661 conn(0x5616e0c18b30 msgr2=0x5616e0c286f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 shutdown_connections 2024-04-23T18:23:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 -- 172.21.15.32:0/3457283661 wait complete. 2024-04-23T18:23:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.762+0000 7fb8e7e58740 1 -- start start 2024-04-23T18:23:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 Processor -- start 2024-04-23T18:23:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c24f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c25460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01aa0a0 con 0x5616e0c1ce80 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01ab510 con 0x5616e0c2bc10 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01aa770 con 0x5616e0c1c560 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8e7e58740 10 client.0 _ll_register_callbacks cb 0x7fb8d4009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39504/0 (socket says 172.21.15.32:39504) 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 -- 172.21.15.32:0/596722337 learned_addr learned my addr 172.21.15.32:0/596722337 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8df59e640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c25460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8de59c640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c24f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 -- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 msgr2=0x5616e0c25460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c25460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 -- 172.21.15.32:0/596722337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 msgr2=0x5616e0c24f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c24f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.763+0000 7fb8ded9d640 1 -- 172.21.15.32:0/596722337 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e0c32d70 con 0x5616e0c1c560 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.764+0000 7fb8ded9d640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8d009c0e0 tx=0x7fb8d009be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=470b1d97b95c63c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.764+0000 7fb8c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:23:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.764+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00b9020 con 0x5616e0c1c560 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.764+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8484+0+0 (secure 0 0 0) 0x7fb8d0098a50 con 0x5616e0c1c560 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.764+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d00a06e0 con 0x5616e0c1c560 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.765+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb8d0020070 con 0x5616e0c1c560 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.765+0000 7fb8e7e58740 10 client.26128 fetch_fsmap learned FSMap version 39 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.765+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5616e0c34040 con 0x5616e0c1c560 2024-04-23T18:23:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d00a06e0 con 0x5616e0c1c560 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7fb8d00981c0 con 0x5616e0c1c560 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 10 client.26128 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 10 client.26128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 20 client.26128 populate_metadata read hostname 'smithi032' 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c1f260 0x5616e0c21650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 4 client.26128 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616e0b89200 con 0x5616e0c1f260 2024-04-23T18:23:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.766+0000 7fb8de59c640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c1f260 0x5616e0c21650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.767+0000 7fb8c2ff5640 10 client.26128 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.767+0000 7fb8de59c640 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c1f260 0x5616e0c21650 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa281881d4256e50 server_cookie=67f3017065fcaee4 in_seq=0 out_seq=0 2024-04-23T18:23:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.783+0000 7fb8c2ff5640 1 -- 172.21.15.32:0/596722337 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616e0b89200 con 0x5616e0c1f260 2024-04-23T18:23:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.783+0000 7fb8c2ff5640 10 client.26128 handle_command_reply: tid=0 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.783+0000 7fb8e7e58740 1 client.26128 shutdown 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.783+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 msgr2=0x5616e0c249e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.783+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8d009c0e0 tx=0x7fb8d009be40 comp rx=0 tx=0).stop 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 shutdown_connections 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5616e0c1f260 0x5616e0c21650 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8c2ff5640 0 client.26128 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5616e0c2bc10 0x5616e0c25460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616e0c1ce80 0x5616e0c24f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 --2- 172.21.15.32:0/596722337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5616e0c1c560 0x5616e0c249e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 >> 172.21.15.32:0/596722337 conn(0x5616e0c18b30 msgr2=0x5616e0c23670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 shutdown_connections 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 1 -- 172.21.15.32:0/596722337 wait complete. 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.784+0000 7fb8e7e58740 20 client.26128 trim_cache size 0 max 16384 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:23:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 msgr2=0x7fb8c80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 0x7fb8c80a6250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004af0 tx=0x7fb8d802a6f0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 msgr2=0x7fb8e0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000d950 tx=0x7fb8d000de10 comp rx=0 tx=0).stop 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 shutdown_connections 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8e0108080 0x7fb8e019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fb8c80a3d90 0x7fb8c80a6250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e006ab00 0x7fb8e0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 --2- 172.21.15.32:0/352959147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e006a140 0x7fb8e0198680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 >> 172.21.15.32:0/352959147 conn(0x7fb8e0077d30 msgr2=0x7fb8e00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 shutdown_connections 2024-04-23T18:23:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:23:58.785+0000 7fb8e513d640 1 -- 172.21.15.32:0/352959147 wait complete. 2024-04-23T18:23:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 510.0 s, 2868.0 tps, lat 34.732 ms stddev 39.594 2024-04-23T18:23:59.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:23:58 smithi032 ceph-mon[29221]: pgmap v273: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.5 MiB/s wr, 582 op/s 2024-04-23T18:23:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:23:58 smithi040 ceph-mon[32626]: pgmap v273: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.5 MiB/s wr, 582 op/s 2024-04-23T18:23:59.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:23:58 smithi064 ceph-mon[33062]: pgmap v273: 97 pgs: 97 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.5 MiB/s wr, 582 op/s 2024-04-23T18:24:01.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:00 smithi032 ceph-mon[29221]: pgmap v274: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:24:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:00 smithi040 ceph-mon[32626]: pgmap v274: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:24:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:00 smithi064 ceph-mon[33062]: pgmap v274: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:24:03.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:02 smithi032 ceph-mon[29221]: pgmap v275: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.1 MiB/s wr, 704 op/s 2024-04-23T18:24:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:02 smithi040 ceph-mon[32626]: pgmap v275: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.1 MiB/s wr, 704 op/s 2024-04-23T18:24:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:02 smithi064 ceph-mon[33062]: pgmap v275: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.1 MiB/s wr, 704 op/s 2024-04-23T18:24:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 515.0 s, 1205.0 tps, lat 81.496 ms stddev 159.579 2024-04-23T18:24:05.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:04 smithi032 ceph-mon[29221]: pgmap v276: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 918 op/s 2024-04-23T18:24:05.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:04 smithi040 ceph-mon[32626]: pgmap v276: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 918 op/s 2024-04-23T18:24:05.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:04 smithi064 ceph-mon[33062]: pgmap v276: 97 pgs: 97 active+clean; 4.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 918 op/s 2024-04-23T18:24:07.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:06 smithi032 ceph-mon[29221]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 30 MiB/s wr, 1.77k op/s 2024-04-23T18:24:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:06 smithi040 ceph-mon[32626]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 30 MiB/s wr, 1.77k op/s 2024-04-23T18:24:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:06 smithi064 ceph-mon[33062]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 30 MiB/s wr, 1.77k op/s 2024-04-23T18:24:08.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:07 smithi032 ceph-mon[29221]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:24:08.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:07 smithi040 ceph-mon[32626]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:24:08.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:07 smithi064 ceph-mon[33062]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:24:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 520.0 s, 2739.2 tps, lat 37.197 ms stddev 53.183 2024-04-23T18:24:09.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:08 smithi032 ceph-mon[29221]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 33 MiB/s wr, 1.85k op/s 2024-04-23T18:24:09.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:08 smithi040 ceph-mon[32626]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 33 MiB/s wr, 1.85k op/s 2024-04-23T18:24:09.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:08 smithi064 ceph-mon[33062]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 33 MiB/s wr, 1.85k op/s 2024-04-23T18:24:11.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:10 smithi032 ceph-mon[29221]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 1.79k op/s 2024-04-23T18:24:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:10 smithi032 ceph-mon[29221]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:24:11.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:10 smithi040 ceph-mon[32626]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 1.79k op/s 2024-04-23T18:24:11.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:10 smithi040 ceph-mon[32626]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:24:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:10 smithi064 ceph-mon[33062]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 1.79k op/s 2024-04-23T18:24:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:10 smithi064 ceph-mon[33062]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:24:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 -- 172.21.15.32:0/2519402483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76701010a0 msgr2=0x7f7670101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 --2- 172.21.15.32:0/2519402483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76701010a0 0x7f7670101480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f766c009900 tx=0x7f766c02a410 comp rx=0 tx=0).stop 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 -- 172.21.15.32:0/2519402483 shutdown_connections 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 --2- 172.21.15.32:0/2519402483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7670106170 0x7f7670069770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 --2- 172.21.15.32:0/2519402483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7670101a50 0x7f7670105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 --2- 172.21.15.32:0/2519402483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76701010a0 0x7f7670101480 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 -- 172.21.15.32:0/2519402483 >> 172.21.15.32:0/2519402483 conn(0x7f7670077f70 msgr2=0x7f7670078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 -- 172.21.15.32:0/2519402483 shutdown_connections 2024-04-23T18:24:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.917+0000 7f7678530640 1 -- 172.21.15.32:0/2519402483 wait complete. 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 Processor -- start 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 -- start start 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76701010a0 0x7f76701a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7670106170 0x7f76701a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76701144e0 con 0x7f767019a880 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7670114360 con 0x7f76701010a0 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7678530640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7670114660 con 0x7f7670106170 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f76762cc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76701010a0 0x7f76701a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7675acb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7670106170 0x7f76701a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47470/0 (socket says 172.21.15.32:47470) 2024-04-23T18:24:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 -- 172.21.15.32:0/1373027512 learned_addr learned my addr 172.21.15.32:0/1373027512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 -- 172.21.15.32:0/1373027512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7670106170 msgr2=0x7f76701a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7670106170 0x7f76701a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 -- 172.21.15.32:0/1373027512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76701010a0 msgr2=0x7f76701a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76701010a0 0x7f76701a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.918+0000 7f7676acd640 1 -- 172.21.15.32:0/1373027512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767019b3a0 con 0x7f767019a880 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.919+0000 7f7676acd640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f766800eb10 tx=0x7f766800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.919+0000 7f76677fe640 1 -- 172.21.15.32:0/1373027512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668004b20 con 0x7f767019a880 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.919+0000 7f76677fe640 1 -- 172.21.15.32:0/1373027512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7668004cc0 con 0x7f767019a880 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.919+0000 7f76677fe640 1 -- 172.21.15.32:0/1373027512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668004b20 con 0x7f767019a880 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.920+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767019b690 con 0x7f767019a880 2024-04-23T18:24:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.920+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76701a78e0 con 0x7f767019a880 2024-04-23T18:24:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 Processor -- start 2024-04-23T18:24:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f76677fe640 1 -- 172.21.15.32:0/1373027512 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7668005020 con 0x7f767019a880 2024-04-23T18:24:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 -- start start 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4f40 0x561cb49bf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49bf800 0x561cb49c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561cb4919970 con 0x561cb49b4f40 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cb48b64a0 con 0x561cb49bf800 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f767b247740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561cb4915930 con 0x561cb49b4620 2024-04-23T18:24:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f7675acb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f7675acb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:38086/0 (socket says 172.21.15.32:38086) 2024-04-23T18:24:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.924+0000 7f7675acb640 1 -- 172.21.15.32:0/891063522 learned_addr learned my addr 172.21.15.32:0/891063522 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f76762cc640 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4f40 0x561cb49bf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49bf800 0x561cb49c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 -- 172.21.15.32:0/891063522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49bf800 msgr2=0x561cb49c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49bf800 0x561cb49c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 -- 172.21.15.32:0/891063522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4f40 msgr2=0x561cb49bf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4f40 0x561cb49bf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f7675acb640 1 -- 172.21.15.32:0/891063522 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb462c210 con 0x561cb49b4620 2024-04-23T18:24:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.925+0000 7f76677fe640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 0x7f765c0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f7675acb640 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7660004650 tx=0x7f7660004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a308aa140f8c1a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f76677fe640 1 -- 172.21.15.32:0/1373027512 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f76680d0930 con 0x7f767019a880 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f76762cc640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 0x7f765c0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f7664bfe640 1 -- 172.21.15.32:0/891063522 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660003c00 con 0x561cb49b4620 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f7664bfe640 1 -- 172.21.15.32:0/891063522 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7660003da0 con 0x561cb49b4620 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.926+0000 7f7664bfe640 1 -- 172.21.15.32:0/891063522 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660011440 con 0x561cb49b4620 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- 172.21.15.32:0/891063522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 msgr2=0x561cb49b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7660004650 tx=0x7f7660004310 comp rx=0 tx=0).stop 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f76762cc640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 0x7f765c0a63d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- 172.21.15.32:0/891063522 shutdown_connections 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49bf800 0x561cb49c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4f40 0x561cb49bf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 --2- 172.21.15.32:0/891063522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49b4a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- 172.21.15.32:0/891063522 >> 172.21.15.32:0/891063522 conn(0x561cb49b0b30 msgr2=0x561cb49b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- 172.21.15.32:0/891063522 shutdown_connections 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- 172.21.15.32:0/891063522 wait complete. 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 -- start start 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.927+0000 7f767b247740 1 Processor -- start 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 0x561cb49bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561cb49c2430 con 0x561cb49bf800 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cb48b40b0 con 0x561cb49b4f40 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561cb49c25b0 con 0x561cb49b4620 2024-04-23T18:24:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 10 client.0 _ll_register_callbacks cb 0x561cb48a1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f76762cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7675acb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 0x561cb49bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7675acb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:57020/0 (socket says 172.21.15.32:57020) 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7675acb640 1 -- 172.21.15.32:0/125792222 learned_addr learned my addr 172.21.15.32:0/125792222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f76762cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:38102/0 (socket says 172.21.15.32:38102) 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 -- 172.21.15.32:0/125792222 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 msgr2=0x561cb49bf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 -- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 msgr2=0x561cb49bc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 -- 172.21.15.32:0/125792222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb46f8620 con 0x561cb49bf800 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7675acb640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f76762cc640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7676acd640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 0x561cb49bc9a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f76680ce0a0 tx=0x7f766809c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b771bd223aac1cd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76680872c0 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f766809cb50 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.928+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7660003da0 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f76680a0ca0 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 10 client.16569 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561cb49ce3c0 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7660003da0 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f7668098640 con 0x561cb49bf800 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 10 client.16569 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 20 client.16569 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb49d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 4 client.16569 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.929+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb4927390 con 0x561cb49cf920 2024-04-23T18:24:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.930+0000 7f76762cc640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb49d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.930+0000 7f76762cc640 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb49d1d10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b3b128579ab343 server_cookie=99b5e95804ffe4a6 in_seq=0 out_seq=0 2024-04-23T18:24:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.930+0000 7f7659ffb640 10 client.16569 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f7659ffb640 1 -- 172.21.15.32:0/125792222 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561cb4927390 con 0x561cb49cf920 2024-04-23T18:24:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f7659ffb640 10 client.16569 handle_command_reply: tid=0 2024-04-23T18:24:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 client.16569 shutdown 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 msgr2=0x561cb49bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 0x561cb49bc9a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f76680ce0a0 tx=0x7f766809c250 comp rx=0 tx=0).stop 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 shutdown_connections 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb49d1d10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f7659ffb640 0 client.16569 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49bf800 0x561cb49bc9a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49bc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 --2- 172.21.15.32:0/125792222 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49b4620 0x561cb49bf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.931+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 >> 172.21.15.32:0/125792222 conn(0x561cb49b0b30 msgr2=0x561cb49b68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.932+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 shutdown_connections 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.932+0000 7f767b247740 1 -- 172.21.15.32:0/125792222 wait complete. 2024-04-23T18:24:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.932+0000 7f767b247740 20 client.16569 trim_cache size 0 max 16384 2024-04-23T18:24:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 Processor -- start 2024-04-23T18:24:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 -- start start 2024-04-23T18:24:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49bec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76701b2680 con 0x561cb49b4620 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76701aff20 con 0x561cb49b4f40 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f767b247740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f76701add90 con 0x561cb49bf800 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47476/0 (socket says 172.21.15.32:47476) 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 -- 172.21.15.32:0/150562353 learned_addr learned my addr 172.21.15.32:0/150562353 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f7675acb640 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f7676acd640 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 -- 172.21.15.32:0/150562353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 msgr2=0x561cb49bec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49bec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 -- 172.21.15.32:0/150562353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 msgr2=0x561cb49afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 -- 172.21.15.32:0/150562353 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb49ba990 con 0x561cb49b4620 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.952+0000 7f76762cc640 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f766c02a930 tx=0x7f766c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b0cd11e80ccc50 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f765bfff640 1 -- 172.21.15.32:0/150562353 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c035040 con 0x561cb49b4620 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f765bfff640 1 -- 172.21.15.32:0/150562353 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f766c058070 con 0x561cb49b4620 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f765bfff640 1 -- 172.21.15.32:0/150562353 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c005550 con 0x561cb49b4620 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f767b247740 1 -- 172.21.15.32:0/150562353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 msgr2=0x561cb49cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f767b247740 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f766c02a930 tx=0x7f766c03f000 comp rx=0 tx=0).stop 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.953+0000 7f767b247740 1 -- 172.21.15.32:0/150562353 shutdown_connections 2024-04-23T18:24:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49bec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 --2- 172.21.15.32:0/150562353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49cb2f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 -- 172.21.15.32:0/150562353 >> 172.21.15.32:0/150562353 conn(0x561cb49b0b30 msgr2=0x561cb49af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 -- 172.21.15.32:0/150562353 shutdown_connections 2024-04-23T18:24:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 -- 172.21.15.32:0/150562353 wait complete. 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.954+0000 7f767b247740 1 -- start start 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 Processor -- start 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49b9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76701ae3e0 con 0x561cb49b4620 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76701af850 con 0x561cb49b4f40 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f76701aeab0 con 0x561cb49bf800 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f767b247740 10 client.0 _ll_register_callbacks cb 0x7f766809c6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47492/0 (socket says 172.21.15.32:47492) 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 -- 172.21.15.32:0/3207056399 learned_addr learned my addr 172.21.15.32:0/3207056399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f7675acb640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f7676acd640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 -- 172.21.15.32:0/3207056399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 msgr2=0x561cb49b9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49b9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 -- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 msgr2=0x561cb49b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.955+0000 7f76762cc640 1 -- 172.21.15.32:0/3207056399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb49b9540 con 0x561cb49b4620 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.956+0000 7f76762cc640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f766c0092a0 tx=0x7f766c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3544f2922e0fcfca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.956+0000 7f7659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.957+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c062070 con 0x561cb49b4620 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.957+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f766c0053f0 con 0x561cb49b4620 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.957+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f766c058070 con 0x561cb49b4620 2024-04-23T18:24:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.958+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f766c06e050 con 0x561cb49b4620 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.958+0000 7f767b247740 10 client.16575 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.958+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561cb4a35b80 con 0x561cb49b4620 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.958+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f766c058070 con 0x561cb49b4620 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.958+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f766c004ab0 con 0x561cb49b4620 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 10 client.16575 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 20 client.16575 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb4a38260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 4 client.16575 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cb4921200 con 0x561cb49cf920 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f7675acb640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb4a38260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f7659ffb640 10 client.16575 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.959+0000 7f7675acb640 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb4a38260 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eeb9ddf41e5c4f2 server_cookie=1889c93919ccb04c in_seq=0 out_seq=0 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f7659ffb640 1 -- 172.21.15.32:0/3207056399 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cb4921200 con 0x561cb49cf920 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f7659ffb640 10 client.16575 handle_command_reply: tid=0 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 client.16575 shutdown 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 msgr2=0x561cb49c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f766c0092a0 tx=0x7f766c005c00 comp rx=0 tx=0).stop 2024-04-23T18:24:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 shutdown_connections 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cb49cf920 0x561cb4a38260 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cb49bf800 0x561cb49b9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cb49b4f40 0x561cb49b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 --2- 172.21.15.32:0/3207056399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cb49b4620 0x561cb49c79f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 >> 172.21.15.32:0/3207056399 conn(0x561cb49b0b30 msgr2=0x561cb49afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.970+0000 7f7659ffb640 0 client.16575 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.971+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 shutdown_connections 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.971+0000 7f767b247740 1 -- 172.21.15.32:0/3207056399 wait complete. 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.971+0000 7f767b247740 20 client.16575 trim_cache size 0 max 16384 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:24:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 msgr2=0x7f765c0a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 0x7f765c0a63d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c031400 comp rx=0 tx=0).stop 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 msgr2=0x7f767019ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f766800eb10 tx=0x7f766800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 shutdown_connections 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f765c0a3f10 0x7f765c0a63d0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c031400 comp rx=0 tx=0).stop 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f767019a880 0x7f767019ad30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7670106170 0x7f76701a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 --2- 172.21.15.32:0/1373027512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76701010a0 0x7f76701a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.972+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 >> 172.21.15.32:0/1373027512 conn(0x7f7670077f70 msgr2=0x7f76700ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.973+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 shutdown_connections 2024-04-23T18:24:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:11.973+0000 7f7678530640 1 -- 172.21.15.32:0/1373027512 wait complete. 2024-04-23T18:24:13.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:12 smithi032 ceph-mon[29221]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 MiB/s wr, 1.92k op/s 2024-04-23T18:24:13.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:12 smithi040 ceph-mon[32626]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 MiB/s wr, 1.92k op/s 2024-04-23T18:24:13.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:12 smithi064 ceph-mon[33062]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 MiB/s wr, 1.92k op/s 2024-04-23T18:24:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 525.0 s, 2885.8 tps, lat 34.886 ms stddev 37.595 2024-04-23T18:24:15.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:14 smithi032 ceph-mon[29221]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 649 op/s 2024-04-23T18:24:15.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:14 smithi040 ceph-mon[32626]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 649 op/s 2024-04-23T18:24:15.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:14 smithi064 ceph-mon[33062]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 649 op/s 2024-04-23T18:24:17.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:16 smithi032 ceph-mon[29221]: pgmap v284: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-23T18:24:17.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:16 smithi040 ceph-mon[32626]: pgmap v284: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-23T18:24:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:16 smithi064 ceph-mon[33062]: pgmap v284: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-23T18:24:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 530.0 s, 2244.6 tps, lat 44.210 ms stddev 46.413 2024-04-23T18:24:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:18 smithi040 ceph-mon[32626]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 916 op/s 2024-04-23T18:24:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:18 smithi064 ceph-mon[33062]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 916 op/s 2024-04-23T18:24:19.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:18 smithi032 ceph-mon[29221]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 916 op/s 2024-04-23T18:24:21.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:20 smithi040 ceph-mon[32626]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 MiB/s wr, 782 op/s 2024-04-23T18:24:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:20 smithi064 ceph-mon[33062]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 MiB/s wr, 782 op/s 2024-04-23T18:24:21.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:20 smithi032 ceph-mon[29221]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 MiB/s wr, 782 op/s 2024-04-23T18:24:23.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:22 smithi040 ceph-mon[32626]: pgmap v287: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 719 op/s 2024-04-23T18:24:23.121 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:22 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:22 smithi064 ceph-mon[33062]: pgmap v287: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 719 op/s 2024-04-23T18:24:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:22 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:23.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:22 smithi032 ceph-mon[29221]: pgmap v287: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 719 op/s 2024-04-23T18:24:23.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:22 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 535.0 s, 2332.8 tps, lat 43.115 ms stddev 50.408 2024-04-23T18:24:24.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:24.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:24.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:24:24.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:24.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:24.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:24:24.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:24.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:24.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:24:25.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:24 smithi040 ceph-mon[32626]: pgmap v288: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 665 op/s 2024-04-23T18:24:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.125+0000 7f9063f22640 1 -- 172.21.15.32:0/201558938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c112c50 msgr2=0x7f905c115040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.125+0000 7f9063f22640 1 --2- 172.21.15.32:0/201558938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c112c50 0x7f905c115040 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9054009900 tx=0x7f905402a3f0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.125+0000 7f9063f22640 1 -- 172.21.15.32:0/201558938 shutdown_connections 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.125+0000 7f9063f22640 1 --2- 172.21.15.32:0/201558938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c112c50 0x7f905c115040 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.126+0000 7f9063f22640 1 --2- 172.21.15.32:0/201558938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 0x7f905c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.126+0000 7f9063f22640 1 --2- 172.21.15.32:0/201558938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c107f00 0x7f905c1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.126+0000 7f9063f22640 1 -- 172.21.15.32:0/201558938 >> 172.21.15.32:0/201558938 conn(0x7f905c075d60 msgr2=0x7f905c076170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.126+0000 7f9063f22640 1 -- 172.21.15.32:0/201558938 shutdown_connections 2024-04-23T18:24:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.126+0000 7f9063f22640 1 -- 172.21.15.32:0/201558938 wait complete. 2024-04-23T18:24:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.127+0000 7f9063f22640 1 Processor -- start 2024-04-23T18:24:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.127+0000 7f9063f22640 1 -- start start 2024-04-23T18:24:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.127+0000 7f9063f22640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 0x7f905c073770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f9063f22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 0x7f905c06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f9063f22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 0x7f905c06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f9063f22640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1187e0 con 0x7f905c107f00 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f90624bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 0x7f905c06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f90614bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 0x7f905c06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f90624bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 0x7f905c06cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:33478/0 (socket says 172.21.15.32:33478) 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f90624bf640 1 -- 172.21.15.32:0/2236010362 learned_addr learned my addr 172.21.15.32:0/2236010362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f905c118660 con 0x7f905c112c50 2024-04-23T18:24:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.128+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f905c118960 con 0x7f905c100920 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f9061cbe640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 0x7f905c073770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 -- 172.21.15.32:0/2236010362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 msgr2=0x7f905c073770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 0x7f905c073770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 -- 172.21.15.32:0/2236010362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 msgr2=0x7f905c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 0x7f905c06cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 -- 172.21.15.32:0/2236010362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c06d2f0 con 0x7f905c107f00 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.129+0000 7f90614bd640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 0x7f905c06c7f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9f0 tx=0x7f904c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.130+0000 7f9052ffd640 1 -- 172.21.15.32:0/2236010362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c00ccc0 con 0x7f905c107f00 2024-04-23T18:24:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.130+0000 7f9052ffd640 1 -- 172.21.15.32:0/2236010362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f904c0108e0 con 0x7f905c107f00 2024-04-23T18:24:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.130+0000 7f9052ffd640 1 -- 172.21.15.32:0/2236010362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c018880 con 0x7f905c107f00 2024-04-23T18:24:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.130+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c06d5e0 con 0x7f905c107f00 2024-04-23T18:24:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.130+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c071550 con 0x7f905c107f00 2024-04-23T18:24:25.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.132+0000 7f9066c31740 1 Processor -- start 2024-04-23T18:24:25.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 -- start start 2024-04-23T18:24:25.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9052ffd640 1 -- 172.21.15.32:0/2236010362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f904c010430 con 0x7f905c107f00 2024-04-23T18:24:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d151a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0b6970 con 0x555d5d151f40 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0534a0 con 0x555d5d15c800 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9066c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0b2930 con 0x555d5d151620 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42606/0 (socket says 172.21.15.32:42606) 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f9061cbe640 1 -- 172.21.15.32:0/2732183899 learned_addr learned my addr 172.21.15.32:0/2732183899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.133+0000 7f90624bf640 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 -- 172.21.15.32:0/2732183899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 msgr2=0x555d5d151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9052ffd640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 0x7f90480a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 -- 172.21.15.32:0/2732183899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 msgr2=0x555d5d15ebf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:24:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 -- 172.21.15.32:0/2732183899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5cdc9210 con 0x555d5d151f40 2024-04-23T18:24:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9052ffd640 1 -- 172.21.15.32:0/2236010362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f904c0d0b10 con 0x7f905c107f00 2024-04-23T18:24:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.134+0000 7f9061cbe640 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9058009f30 tx=0x7f905800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718d82e3bce43ea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.135+0000 7f9047fff640 1 -- 172.21.15.32:0/2732183899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058004d20 con 0x555d5d151f40 2024-04-23T18:24:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.135+0000 7f9047fff640 1 -- 172.21.15.32:0/2732183899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90580054d0 con 0x555d5d151f40 2024-04-23T18:24:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.135+0000 7f9047fff640 1 -- 172.21.15.32:0/2732183899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058011970 con 0x555d5d151f40 2024-04-23T18:24:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.135+0000 7f90624bf640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 0x7f90480a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.136+0000 7f90624bf640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 0x7f90480a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90540096c0 tx=0x7f9054005a40 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.136+0000 7f9066c31740 1 -- 172.21.15.32:0/2732183899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 msgr2=0x555d5d15c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.136+0000 7f9066c31740 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9058009f30 tx=0x7f905800f7f0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 -- 172.21.15.32:0/2732183899 shutdown_connections 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d15c2c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 --2- 172.21.15.32:0/2732183899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 -- 172.21.15.32:0/2732183899 >> 172.21.15.32:0/2732183899 conn(0x555d5d14db30 msgr2=0x555d5d14df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 -- 172.21.15.32:0/2732183899 shutdown_connections 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 -- 172.21.15.32:0/2732183899 wait complete. 2024-04-23T18:24:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.137+0000 7f9066c31740 1 -- start start 2024-04-23T18:24:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.138+0000 7f9066c31740 1 Processor -- start 2024-04-23T18:24:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d15c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555d5d15f430 con 0x555d5d151f40 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0510b0 con 0x555d5d15c800 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555d5d15f5b0 con 0x555d5d151620 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d15c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9066c31740 10 client.0 _ll_register_callbacks cb 0x555d5d03ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d15c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40046/0 (socket says 172.21.15.32:40046) 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f9061cbe640 1 -- 172.21.15.32:0/1693863113 learned_addr learned my addr 172.21.15.32:0/1693863113 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f90624bf640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.140+0000 7f90614bd640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 -- 172.21.15.32:0/1693863113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 msgr2=0x555d5d15c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d15c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 -- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 msgr2=0x555d5d159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 -- 172.21.15.32:0/1693863113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5ce95620 con 0x555d5d151f40 2024-04-23T18:24:25.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.141+0000 7f90614bd640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d159520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ce280 tx=0x7f904c010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d026562fc9b39c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.142+0000 7f90457fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.142+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c0a0e80 con 0x555d5d151f40 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.142+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f904c09add0 con 0x555d5d151f40 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.142+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90580054d0 con 0x555d5d151f40 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f904c0a5740 con 0x555d5d151f40 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 10 client.16587 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555d5d16b410 con 0x555d5d151f40 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90580054d0 con 0x555d5d151f40 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f904c0989f0 con 0x555d5d151f40 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 10 client.16587 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 20 client.16587 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.143+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d16ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.144+0000 7f9066c31740 4 client.16587 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.144+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d5d0c4390 con 0x555d5d16c880 2024-04-23T18:24:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.144+0000 7f9061cbe640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d16ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.145+0000 7f9061cbe640 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d16ec70 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d796734c4e96b36 server_cookie=9aa340c43d57b077 in_seq=0 out_seq=0 2024-04-23T18:24:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.145+0000 7f90457fa640 10 client.16587 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.145+0000 7f90457fa640 1 -- 172.21.15.32:0/1693863113 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d5d0c4390 con 0x555d5d16c880 2024-04-23T18:24:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.145+0000 7f90457fa640 10 client.16587 handle_command_reply: tid=0 2024-04-23T18:24:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.146+0000 7f9066c31740 1 client.16587 shutdown 2024-04-23T18:24:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.147+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 msgr2=0x555d5d159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.147+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d159520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ce280 tx=0x7f904c010720 comp rx=0 tx=0).stop 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 shutdown_connections 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d16ec70 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f90457fa640 0 client.16587 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151f40 0x555d5d159520 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 --2- 172.21.15.32:0/1693863113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d15c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 >> 172.21.15.32:0/1693863113 conn(0x555d5d14db30 msgr2=0x555d5d14ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 shutdown_connections 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 1 -- 172.21.15.32:0/1693863113 wait complete. 2024-04-23T18:24:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.148+0000 7f9066c31740 20 client.16587 trim_cache size 0 max 16384 2024-04-23T18:24:25.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.177+0000 7f9066c31740 1 Processor -- start 2024-04-23T18:24:25.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.177+0000 7f9066c31740 1 -- start start 2024-04-23T18:24:25.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151f40 0x555d5d14cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:24 smithi064 ceph-mon[33062]: pgmap v288: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 665 op/s 2024-04-23T18:24:25.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42622/0 (socket says 172.21.15.32:42622) 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f90614bd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151f40 0x555d5d14cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 -- 172.21.15.32:0/3584333411 learned_addr learned my addr 172.21.15.32:0/3584333411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1bf290 con 0x555d5d151620 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1bcb30 con 0x555d5d15c800 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1ba9a0 con 0x555d5d151f40 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f90624bf640 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 -- 172.21.15.32:0/3584333411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151f40 msgr2=0x555d5d14cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151f40 0x555d5d14cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 -- 172.21.15.32:0/3584333411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 msgr2=0x555d5d15bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.178+0000 7f9061cbe640 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.179+0000 7f9061cbe640 1 -- 172.21.15.32:0/3584333411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5d1528e0 con 0x555d5d151620 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.179+0000 7f9061cbe640 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9058011590 tx=0x7f9058005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b99cca86e3022a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.179+0000 7f90477fe640 1 -- 172.21.15.32:0/3584333411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058003b70 con 0x555d5d151620 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.179+0000 7f90477fe640 1 -- 172.21.15.32:0/3584333411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9058003d10 con 0x555d5d151620 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.179+0000 7f90477fe640 1 -- 172.21.15.32:0/3584333411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058011680 con 0x555d5d151620 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.180+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 msgr2=0x555d5d1682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.180+0000 7f9066c31740 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9058011590 tx=0x7f9058005fd0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.181+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 shutdown_connections 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.181+0000 7f9066c31740 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d15c800 0x555d5d15bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.181+0000 7f9066c31740 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151f40 0x555d5d14cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.181+0000 7f9066c31740 1 --2- 172.21.15.32:0/3584333411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d151620 0x555d5d1682f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.181+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 >> 172.21.15.32:0/3584333411 conn(0x555d5d14db30 msgr2=0x555d5d14c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.182+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 shutdown_connections 2024-04-23T18:24:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.182+0000 7f9066c31740 1 -- 172.21.15.32:0/3584333411 wait complete. 2024-04-23T18:24:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.182+0000 7f9066c31740 1 -- start start 2024-04-23T18:24:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.182+0000 7f9066c31740 1 Processor -- start 2024-04-23T18:24:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d164c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 0x555d5d15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 0x555d5d160550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1baff0 con 0x555d5d15c800 2024-04-23T18:24:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1bc460 con 0x555d5d151f40 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1bb6c0 con 0x555d5d151620 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d164c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9061cbe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d164c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40064/0 (socket says 172.21.15.32:40064) 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9061cbe640 1 -- 172.21.15.32:0/1253787282 learned_addr learned my addr 172.21.15.32:0/1253787282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f90614bd640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 0x555d5d15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f9066c31740 10 client.0 _ll_register_callbacks cb 0x7f904c09c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.183+0000 7f90624bf640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 0x555d5d160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90624bf640 1 -- 172.21.15.32:0/1253787282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 msgr2=0x555d5d164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90624bf640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d164c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90624bf640 1 -- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 msgr2=0x555d5d15ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90624bf640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 0x555d5d15ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90624bf640 1 -- 172.21.15.32:0/1253787282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5d160d80 con 0x555d5d15c800 2024-04-23T18:24:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.184+0000 7f90614bd640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 0x555d5d15ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.185+0000 7f90624bf640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 0x555d5d160550 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9054031440 tx=0x7f9054009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ab4e679aa75902 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.185+0000 7f90457fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.185+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054035040 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.185+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f90540314a0 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.185+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9058003d10 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f90540316a0 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 10 client.16593 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555d5d1d2d90 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9058003d10 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 8342+0+0 (secure 0 0 0) 0x7f9054044070 con 0x555d5d15c800 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 10 client.16593 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 20 client.16593 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d1d54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 4 client.16593 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.186+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d5d14d980 con 0x555d5d16c880 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.187+0000 7f9061cbe640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d1d54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.187+0000 7f90457fa640 10 client.16593 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.188+0000 7f9061cbe640 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d1d54c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7611447af7bb2e server_cookie=7b8a056b56ae613 in_seq=0 out_seq=0 2024-04-23T18:24:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.202+0000 7f90457fa640 1 -- 172.21.15.32:0/1253787282 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d5d14d980 con 0x555d5d16c880 2024-04-23T18:24:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.202+0000 7f90457fa640 10 client.16593 handle_command_reply: tid=0 2024-04-23T18:24:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.202+0000 7f9066c31740 1 client.16593 shutdown 2024-04-23T18:24:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 msgr2=0x555d5d160550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 0x555d5d160550 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9054031440 tx=0x7f9054009260 comp rx=0 tx=0).stop 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 shutdown_connections 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555d5d16c880 0x555d5d1d54c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d5d15c800 0x555d5d160550 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f90457fa640 0 client.16593 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d5d151f40 0x555d5d15ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 --2- 172.21.15.32:0/1253787282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555d5d151620 0x555d5d164c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 >> 172.21.15.32:0/1253787282 conn(0x555d5d14db30 msgr2=0x555d5d14ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 shutdown_connections 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 1 -- 172.21.15.32:0/1253787282 wait complete. 2024-04-23T18:24:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.203+0000 7f9066c31740 20 client.16593 trim_cache size 0 max 16384 2024-04-23T18:24:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:24:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:24:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:24:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:24:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.204+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 msgr2=0x7f90480a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.204+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 0x7f90480a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90540096c0 tx=0x7f9054005a40 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.204+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 msgr2=0x7f905c06c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.204+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 0x7f905c06c7f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9f0 tx=0x7f904c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 shutdown_connections 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f905c112c50 0x7f905c06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f90480a3d90 0x7f90480a6250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f905c107f00 0x7f905c06c7f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 --2- 172.21.15.32:0/2236010362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f905c100920 0x7f905c073770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 >> 172.21.15.32:0/2236010362 conn(0x7f905c075d60 msgr2=0x7f905c111b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 shutdown_connections 2024-04-23T18:24:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:25.205+0000 7f9063f22640 1 -- 172.21.15.32:0/2236010362 wait complete. 2024-04-23T18:24:25.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:24 smithi032 ceph-mon[29221]: pgmap v288: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 665 op/s 2024-04-23T18:24:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:26 smithi040 ceph-mon[32626]: pgmap v289: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 870 op/s 2024-04-23T18:24:27.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:26 smithi064 ceph-mon[33062]: pgmap v289: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 870 op/s 2024-04-23T18:24:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:26 smithi032 ceph-mon[29221]: pgmap v289: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 870 op/s 2024-04-23T18:24:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 540.0 s, 1917.4 tps, lat 52.015 ms stddev 90.725 2024-04-23T18:24:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:28 smithi040 ceph-mon[32626]: pgmap v290: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-23T18:24:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:28 smithi064 ceph-mon[33062]: pgmap v290: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-23T18:24:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:28 smithi032 ceph-mon[29221]: pgmap v290: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-23T18:24:31.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:30 smithi040 ceph-mon[32626]: pgmap v291: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 2.06k op/s 2024-04-23T18:24:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:30 smithi064 ceph-mon[33062]: pgmap v291: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 2.06k op/s 2024-04-23T18:24:31.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:30 smithi032 ceph-mon[29221]: pgmap v291: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 2.06k op/s 2024-04-23T18:24:31.291 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-04-23T18:24:31.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 -- 172.21.15.32:0/1672183841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 msgr2=0x7f1a741072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/1672183841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a741072b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009900 tx=0x7f1a7002a3f0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 -- 172.21.15.32:0/1672183841 shutdown_connections 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/1672183841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a74101a40 0x7f1a7410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/1672183841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a741010a0 0x7f1a74101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/1672183841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a741072b0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 -- 172.21.15.32:0/1672183841 >> 172.21.15.32:0/1672183841 conn(0x7f1a74077c40 msgr2=0x7f1a74078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 -- 172.21.15.32:0/1672183841 shutdown_connections 2024-04-23T18:24:31.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.448+0000 7f1a7bf08640 1 -- 172.21.15.32:0/1672183841 wait complete. 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 Processor -- start 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 -- start start 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 0x7f1a74198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74069860 con 0x7f1a74106ed0 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a740696e0 con 0x7f1a741010a0 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.449+0000 7f1a7bf08640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a740699e0 con 0x7f1a74101a40 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a7a4a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a7a4a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44430/0 (socket says 172.21.15.32:44430) 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a7a4a5640 1 -- 172.21.15.32:0/2989469019 learned_addr learned my addr 172.21.15.32:0/2989469019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:41620/0 (socket says 172.21.15.32:41620) 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a794a3640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 0x7f1a74198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 -- 172.21.15.32:0/2989469019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 msgr2=0x7f1a74198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 0x7f1a74198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 -- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 msgr2=0x7f1a7419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 -- 172.21.15.32:0/2989469019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7419d6d0 con 0x7f1a741010a0 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a7a4a5640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a794a3640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 0x7f1a74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:31.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a79ca4640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1a70000c00 tx=0x7f1a70004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.450+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7003f070 con 0x7f1a741010a0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.451+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7419d960 con 0x7f1a741010a0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.451+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a70003aa0 con 0x7f1a741010a0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.451+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70035070 con 0x7f1a741010a0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.451+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a74110a70 con 0x7f1a741010a0 2024-04-23T18:24:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.451+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a741025b0 con 0x7f1a741010a0 2024-04-23T18:24:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.453+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1a700035f0 con 0x7f1a741010a0 2024-04-23T18:24:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.453+0000 7f1a6affd640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 0x7f1a600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.453+0000 7f1a794a3640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 0x7f1a600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.453+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f1a700f0f80 con 0x7f1a741010a0 2024-04-23T18:24:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.454+0000 7f1a794a3640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 0x7f1a600a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1a74106aa0 tx=0x7f1a64005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.457+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a700bcf90 con 0x7f1a741010a0 2024-04-23T18:24:31.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.853+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a74068e00 con 0x7f1a741010a0 2024-04-23T18:24:31.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.854+0000 7f1a6affd640 1 -- 172.21.15.32:0/2989469019 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11851 (secure 0 0 0) 0x7f1a700c1290 con 0x7f1a741010a0 2024-04-23T18:24:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:24:31.847 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:24:30.869819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:24:31.848 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.856+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 msgr2=0x7f1a600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.856+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 0x7f1a600a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1a74106aa0 tx=0x7f1a64005f90 comp rx=0 tx=0).stop 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.856+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 msgr2=0x7f1a741986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.856+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1a70000c00 tx=0x7f1a70004610 comp rx=0 tx=0).stop 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 shutdown_connections 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1a600a3d20 0x7f1a600a61e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a74106ed0 0x7f1a7419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a74101a40 0x7f1a74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 --2- 172.21.15.32:0/2989469019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a741010a0 0x7f1a741986e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 >> 172.21.15.32:0/2989469019 conn(0x7f1a74077c40 msgr2=0x7f1a740ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 shutdown_connections 2024-04-23T18:24:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:31.857+0000 7f1a7bf08640 1 -- 172.21.15.32:0/2989469019 wait complete. 2024-04-23T18:24:31.866 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 77.7 --await-for 42.0' 2024-04-23T18:24:31.866 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 77.7 --await-for 42.0 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.037+0000 7f5297f43640 1 -- 172.21.15.32:0/3399466422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529010e600 msgr2=0x7f529010e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.037+0000 7f5297f43640 1 --2- 172.21.15.32:0/3399466422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529010e600 0x7f529010e9e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5280009930 tx=0x7f528002a440 comp rx=0 tx=0).stop 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.037+0000 7f5297f43640 1 -- 172.21.15.32:0/3399466422 shutdown_connections 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.037+0000 7f5297f43640 1 --2- 172.21.15.32:0/3399466422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108f00 0x7f52901153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.038+0000 7f5297f43640 1 --2- 172.21.15.32:0/3399466422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5290108540 0x7f52901089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.038+0000 7f5297f43640 1 --2- 172.21.15.32:0/3399466422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529010e600 0x7f529010e9e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.038+0000 7f5297f43640 1 -- 172.21.15.32:0/3399466422 >> 172.21.15.32:0/3399466422 conn(0x7f52900764e0 msgr2=0x7f5290078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.039+0000 7f5297f43640 1 -- 172.21.15.32:0/3399466422 shutdown_connections 2024-04-23T18:24:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.039+0000 7f5297f43640 1 -- 172.21.15.32:0/3399466422 wait complete. 2024-04-23T18:24:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.040+0000 7f5297f43640 1 Processor -- start 2024-04-23T18:24:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.040+0000 7f5297f43640 1 -- start start 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5290108f00 0x7f529019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010e600 0x7f52901a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5295cdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5295cdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:41638/0 (socket says 172.21.15.32:41638) 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5295cdf640 1 -- 172.21.15.32:0/1858926523 learned_addr learned my addr 172.21.15.32:0/1858926523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5290118840 con 0x7f529010e600 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f52954de640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5290108f00 0x7f529019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52901186c0 con 0x7f5290108540 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f52964e0640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010e600 0x7f52901a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.041+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f52901189c0 con 0x7f5290108f00 2024-04-23T18:24:32.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 -- 172.21.15.32:0/1858926523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5290108f00 msgr2=0x7f529019f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5290108f00 0x7f529019f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 -- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010e600 msgr2=0x7f52901a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010e600 0x7f52901a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52901a3d30 con 0x7f5290108540 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5295cdf640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5280031440 tx=0x7f5280003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5286ffd640 1 -- 172.21.15.32:0/1858926523 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528003f070 con 0x7f5290108540 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5286ffd640 1 -- 172.21.15.32:0/1858926523 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5280003e40 con 0x7f5290108540 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52901a3fc0 con 0x7f5290108540 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5286ffd640 1 -- 172.21.15.32:0/1858926523 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280035030 con 0x7f5290108540 2024-04-23T18:24:32.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.042+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52901ab8b0 con 0x7f5290108540 2024-04-23T18:24:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.046+0000 7f5286ffd640 1 -- 172.21.15.32:0/1858926523 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5280042070 con 0x7f5290108540 2024-04-23T18:24:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.046+0000 7f529ac56740 1 Processor -- start 2024-04-23T18:24:32.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 -- start start 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 0x56273683e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f5286ffd640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 0x7f527c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e9c0 0x562736848b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x56273684b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5627367a3370 con 0x56273683e0a0 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5627367402e0 con 0x562736849090 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f529ac56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56273679f330 con 0x56273683e9c0 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f5286ffd640 1 -- 172.21.15.32:0/1858926523 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f52800f0f80 con 0x7f5290108540 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f52954de640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e9c0 0x562736848b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f52964e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x56273684b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.047+0000 7f52964e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 0x56273683e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 0x56273683e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44438/0 (socket says 172.21.15.32:44438) 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 -- 172.21.15.32:0/1684623373 learned_addr learned my addr 172.21.15.32:0/1684623373 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f5295cdf640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 0x7f527c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 -- 172.21.15.32:0/1684623373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e9c0 msgr2=0x562736848b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e9c0 0x562736848b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 -- 172.21.15.32:0/1684623373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 msgr2=0x56273683e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 0x56273683e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 -- 172.21.15.32:0/1684623373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562736581d80 con 0x562736849090 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f52964e0640 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x56273684b480 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5288012ac0 tx=0x7f5288004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1cebdbf9107d47 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f5295cdf640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 0x7f527c0a6320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52800ee6f0 tx=0x7f5280031f10 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f527bfff640 1 -- 172.21.15.32:0/1684623373 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800f820 con 0x562736849090 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f527bfff640 1 -- 172.21.15.32:0/1684623373 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52880103d0 con 0x562736849090 2024-04-23T18:24:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.048+0000 7f527bfff640 1 -- 172.21.15.32:0/1684623373 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528801e620 con 0x562736849090 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.049+0000 7f529ac56740 1 -- 172.21.15.32:0/1684623373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 msgr2=0x56273684b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.049+0000 7f529ac56740 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x56273684b480 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5288012ac0 tx=0x7f5288004a40 comp rx=0 tx=0).stop 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.049+0000 7f529ac56740 1 -- 172.21.15.32:0/1684623373 shutdown_connections 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.049+0000 7f529ac56740 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x56273684b480 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.049+0000 7f529ac56740 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e9c0 0x562736848b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.050+0000 7f529ac56740 1 --2- 172.21.15.32:0/1684623373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e0a0 0x56273683e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.050+0000 7f529ac56740 1 -- 172.21.15.32:0/1684623373 >> 172.21.15.32:0/1684623373 conn(0x56273683a3c0 msgr2=0x56273683a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.050+0000 7f529ac56740 1 -- 172.21.15.32:0/1684623373 shutdown_connections 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.050+0000 7f529ac56740 1 -- 172.21.15.32:0/1684623373 wait complete. 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.051+0000 7f529ac56740 1 -- start start 2024-04-23T18:24:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.051+0000 7f529ac56740 1 Processor -- start 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x5627368464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562736849090 0x562736846a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56273685ce90 con 0x56273683e9c0 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56273673dfb0 con 0x56273683e0a0 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56273685d010 con 0x562736849090 2024-04-23T18:24:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f529ac56740 10 client.0 _ll_register_callbacks cb 0x56273673d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:41660/0 (socket says 172.21.15.32:41660) 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 -- 172.21.15.32:0/1621182713 learned_addr learned my addr 172.21.15.32:0/1621182713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562736849090 0x562736846a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 -- 172.21.15.32:0/1621182713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562736849090 msgr2=0x562736846a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562736849090 0x562736846a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 -- 172.21.15.32:0/1621182713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 msgr2=0x5627368464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x5627368464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.052+0000 7f52964e0640 1 -- 172.21.15.32:0/1621182713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627365e2510 con 0x56273683e0a0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f52964e0640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5288011010 tx=0x7f5288005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e22637baedb5189 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f52797fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288028040 con 0x56273683e0a0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f5288012030 con 0x56273683e0a0 2024-04-23T18:24:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52880103d0 con 0x56273683e0a0 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5288035050 con 0x56273683e0a0 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f529ac56740 10 client.26171 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.053+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562736475670 con 0x56273683e0a0 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5288012230 con 0x56273683e0a0 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f528800f040 con 0x56273683e0a0 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 10 client.26171 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 10 client.26171 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 20 client.26171 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x56273686c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 4 client.26171 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f52880103d0 con 0x56273686a2e0 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.054+0000 7f52954de640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x56273686c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.055+0000 7f52797fa640 10 client.26171 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.055+0000 7f52954de640 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x56273686c6d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8469598e2f75370 server_cookie=5b40ea5f98b443f4 in_seq=0 out_seq=0 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f52797fa640 1 -- 172.21.15.32:0/1621182713 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f52880103d0 con 0x56273686a2e0 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f52797fa640 10 client.26171 handle_command_reply: tid=0 2024-04-23T18:24:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 client.26171 shutdown 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 msgr2=0x562736845fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5288011010 tx=0x7f5288005ac0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 shutdown_connections 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x56273686c6d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562736849090 0x562736846a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f52797fa640 0 client.26171 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x5627368464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 --2- 172.21.15.32:0/1621182713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56273683e0a0 0x562736845fb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 >> 172.21.15.32:0/1621182713 conn(0x56273683a3c0 msgr2=0x5627368400c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 shutdown_connections 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 1 -- 172.21.15.32:0/1621182713 wait complete. 2024-04-23T18:24:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.056+0000 7f529ac56740 20 client.26171 trim_cache size 0 max 16384 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.081+0000 7f529ac56740 1 Processor -- start 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.081+0000 7f529ac56740 1 -- start start 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.081+0000 7f529ac56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.081+0000 7f529ac56740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736839d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f529ac56740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736865d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f529ac56740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f52901ae2a0 con 0x56273683e9c0 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f529ac56740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52901abb70 con 0x562736849090 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f529ac56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5290080f00 con 0x56273683e0a0 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f52964e0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f52964e0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40130/0 (socket says 172.21.15.32:40130) 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f52964e0640 1 -- 172.21.15.32:0/4137271391 learned_addr learned my addr 172.21.15.32:0/4137271391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.082+0000 7f52954de640 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736839d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736865d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 -- 172.21.15.32:0/4137271391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 msgr2=0x562736865d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736865d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 -- 172.21.15.32:0/4137271391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 msgr2=0x562736839d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736839d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 -- 172.21.15.32:0/4137271391 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562736844c00 con 0x56273683e0a0 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f52964e0640 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5288010c60 tx=0x7f52880126c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bb1038635fa4d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f527b7fe640 1 -- 172.21.15.32:0/4137271391 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288010d00 con 0x56273683e0a0 2024-04-23T18:24:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f527b7fe640 1 -- 172.21.15.32:0/4137271391 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5288010870 con 0x56273683e0a0 2024-04-23T18:24:32.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.083+0000 7f527b7fe640 1 -- 172.21.15.32:0/4137271391 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800f7f0 con 0x56273683e0a0 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- 172.21.15.32:0/4137271391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 msgr2=0x562736848710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5288010c60 tx=0x7f52880126c0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- 172.21.15.32:0/4137271391 shutdown_connections 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736865d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736839d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 --2- 172.21.15.32:0/4137271391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736848710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- 172.21.15.32:0/4137271391 >> 172.21.15.32:0/4137271391 conn(0x56273683a3c0 msgr2=0x562736865660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- 172.21.15.32:0/4137271391 shutdown_connections 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- 172.21.15.32:0/4137271391 wait complete. 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.084+0000 7f529ac56740 1 -- start start 2024-04-23T18:24:32.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 Processor -- start 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736843140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736843680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5290081550 con 0x56273683e9c0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52900829c0 con 0x562736849090 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5290081c20 con 0x56273683e0a0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f529ac56740 10 client.0 _ll_register_callbacks cb 0x7f5288010ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f52954de640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736843140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f52964e0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f52964e0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:40142/0 (socket says 172.21.15.32:40142) 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f52964e0640 1 -- 172.21.15.32:0/151974124 learned_addr learned my addr 172.21.15.32:0/151974124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.085+0000 7f52964e0640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736843680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 -- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 msgr2=0x562736843680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736843680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 -- 172.21.15.32:0/151974124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 msgr2=0x562736843140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736843140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 -- 172.21.15.32:0/151974124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627368cf700 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52964e0640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5288005d10 tx=0x7f52880118c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb30464510171ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52797fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880103d0 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5288010870 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f5288009db0 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f52880114e0 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f529ac56740 10 client.26188 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5627368d08b0 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.086+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5288003e20 con 0x56273683e0a0 2024-04-23T18:24:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f5288011d60 con 0x56273683e0a0 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 10 client.26188 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 10 client.26188 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 20 client.26188 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x5627368d2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 4 client.26188 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.7, "await_for": 42.0}] 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.7, "await_for": 42.0}) -- 0x7f5288010870 con 0x56273686a2e0 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f52954de640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x5627368d2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.087+0000 7f52797fa640 10 client.26188 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:32.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:32.088+0000 7f52954de640 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x5627368d2ec0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af2581a252306152 server_cookie=53e301f2e99f45e3 in_seq=0 out_seq=0 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f52797fa640 1 -- 172.21.15.32:0/151974124 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f5288010870 con 0x56273686a2e0 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f52797fa640 10 client.26188 handle_command_reply: tid=0 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 client.26188 shutdown 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 msgr2=0x562736842c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5288005d10 tx=0x7f52880118c0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 shutdown_connections 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56273686a2e0 0x5627368d2ec0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562736849090 0x562736843680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56273683e9c0 0x562736843140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 --2- 172.21.15.32:0/151974124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56273683e0a0 0x562736842c00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f52797fa640 0 client.26188 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 >> 172.21.15.32:0/151974124 conn(0x56273683a3c0 msgr2=0x562736864dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 shutdown_connections 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 1 -- 172.21.15.32:0/151974124 wait complete. 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f529ac56740 20 client.26188 trim_cache size 0 max 16384 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 40, 2024-04-23T18:24:33.011 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 30, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "d0181b4e": { 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 30, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 77.7, 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:24:33.012 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:33.013 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 msgr2=0x7f527c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 0x7f527c0a6320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52800ee6f0 tx=0x7f5280031f10 comp rx=0 tx=0).stop 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 msgr2=0x7f529019ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5280031440 tx=0x7f5280003cd0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 shutdown_connections 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f527c0a3e60 0x7f527c0a6320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010e600 0x7f52901a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5290108f00 0x7f529019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 --2- 172.21.15.32:0/1858926523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290108540 0x7f529019ed40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 >> 172.21.15.32:0/1858926523 conn(0x7f52900764e0 msgr2=0x7f5290077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 shutdown_connections 2024-04-23T18:24:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:33.017+0000 7f5297f43640 1 -- 172.21.15.32:0/1858926523 wait complete. 2024-04-23T18:24:33.021 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd0181b4e', quiesce took 1.7 seconds. Will release after: 17.7 2024-04-23T18:24:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:32 smithi040 ceph-mon[32626]: pgmap v292: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 MiB/s wr, 1.79k op/s 2024-04-23T18:24:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:32 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2989469019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:32 smithi064 ceph-mon[33062]: pgmap v292: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 MiB/s wr, 1.79k op/s 2024-04-23T18:24:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:32 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2989469019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:32 smithi032 ceph-mon[29221]: pgmap v292: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 MiB/s wr, 1.79k op/s 2024-04-23T18:24:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:32 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:33.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2989469019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:33.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 545.0 s, 1898.8 tps, lat 41.738 ms stddev 52.706 2024-04-23T18:24:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:34 smithi040 ceph-mon[32626]: pgmap v293: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-23T18:24:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:34 smithi064 ceph-mon[33062]: pgmap v293: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-23T18:24:35.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:34 smithi032 ceph-mon[29221]: pgmap v293: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-23T18:24:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:36 smithi040 ceph-mon[32626]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 MiB/s wr, 2.04k op/s 2024-04-23T18:24:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:36 smithi064 ceph-mon[33062]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 MiB/s wr, 2.04k op/s 2024-04-23T18:24:37.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:36 smithi032 ceph-mon[29221]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 MiB/s wr, 2.04k op/s 2024-04-23T18:24:38.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- 172.21.15.32:0/2702037997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd81021c0 msgr2=0x7f4fd81025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2702037997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd81021c0 0x7f4fd81025a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009900 tx=0x7f4fd402a410 comp rx=0 tx=0).stop 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- 172.21.15.32:0/2702037997 shutdown_connections 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2702037997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd8107290 0x7f4fd810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2702037997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd8102b70 0x7f4fd8106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2702037997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd81021c0 0x7f4fd81025a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- 172.21.15.32:0/2702037997 >> 172.21.15.32:0/2702037997 conn(0x7f4fd80fda40 msgr2=0x7f4fd80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- 172.21.15.32:0/2702037997 shutdown_connections 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- 172.21.15.32:0/2702037997 wait complete. 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 Processor -- start 2024-04-23T18:24:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.312+0000 7f4fe0635640 1 -- start start 2024-04-23T18:24:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd81021c0 0x7f4fd8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd819d000 0x7f4fd819d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8112250 con 0x7f4fd8107290 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81120d0 con 0x7f4fd819d000 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fe0635640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81123d0 con 0x7f4fd81021c0 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44520/0 (socket says 172.21.15.32:44520) 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2655388233 learned_addr learned my addr 172.21.15.32:0/2655388233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd81021c0 0x7f4fd8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd819d000 0x7f4fd819d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2655388233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd81021c0 msgr2=0x7f4fd8198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd81021c0 0x7f4fd8198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2655388233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd819d000 msgr2=0x7f4fd819d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd819d000 0x7f4fd819d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2655388233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd819daf0 con 0x7f4fd8107290 2024-04-23T18:24:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.313+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c970 tx=0x7f4fc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.314+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2655388233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8014070 con 0x7f4fd8107290 2024-04-23T18:24:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.314+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2655388233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fc8004440 con 0x7f4fd8107290 2024-04-23T18:24:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.314+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd819dde0 con 0x7f4fd8107290 2024-04-23T18:24:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.314+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2655388233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8003e50 con 0x7f4fd8107290 2024-04-23T18:24:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.314+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd81a55d0 con 0x7f4fd8107290 2024-04-23T18:24:38.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.316+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2655388233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4fc8017070 con 0x7f4fd8107290 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 Processor -- start 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 -- start start 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763ff40 0x55e0a764a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a764cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e0a75a4970 con 0x55e0a763f620 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e0a75414a0 con 0x55e0a763ff40 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fe342d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e0a75a0930 con 0x55e0a764a800 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fcf7fe640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 0x7f4fc40a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fdebd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fdebd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44534/0 (socket says 172.21.15.32:44534) 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fdebd2640 1 -- 172.21.15.32:0/175612714 learned_addr learned my addr 172.21.15.32:0/175612714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 0x7f4fc40a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763ff40 0x55e0a764a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2655388233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f4fc80d03d0 con 0x7f4fd8107290 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.317+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a764cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 -- 172.21.15.32:0/175612714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 msgr2=0x55e0a764cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a764cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 -- 172.21.15.32:0/175612714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763ff40 msgr2=0x55e0a764a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763ff40 0x55e0a764a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 -- 172.21.15.32:0/175612714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a72b7210 con 0x55e0a763f620 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00094d0 tx=0x7f4fd00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc2213cc7224f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 0x7f4fc40a6370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0005c40 tx=0x7f4fd001d000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fccbfe640 1 -- 172.21.15.32:0/175612714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0012840 con 0x55e0a763f620 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fccbfe640 1 -- 172.21.15.32:0/175612714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0009d60 con 0x55e0a763f620 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.318+0000 7f4fccbfe640 1 -- 172.21.15.32:0/175612714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000de00 con 0x55e0a763f620 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- 172.21.15.32:0/175612714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 msgr2=0x55e0a763fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00094d0 tx=0x7f4fd00107a0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- 172.21.15.32:0/175612714 shutdown_connections 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a764cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763ff40 0x55e0a764a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 --2- 172.21.15.32:0/175612714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a763fa00 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- 172.21.15.32:0/175612714 >> 172.21.15.32:0/175612714 conn(0x55e0a763bb30 msgr2=0x55e0a763bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- 172.21.15.32:0/175612714 shutdown_connections 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- 172.21.15.32:0/175612714 wait complete. 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.319+0000 7f4fe342d740 1 -- start start 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 Processor -- start 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7654a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a764a800 0x55e0a764e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e0a764d3d0 con 0x55e0a763f620 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e0a753f0b0 con 0x55e0a764a800 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e0a764d550 con 0x55e0a763ff40 2024-04-23T18:24:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fe342d740 10 client.0 _ll_register_callbacks cb 0x55e0a752cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44536/0 (socket says 172.21.15.32:44536) 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 -- 172.21.15.32:0/3313709706 learned_addr learned my addr 172.21.15.32:0/3313709706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a764a800 0x55e0a764e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7654a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 -- 172.21.15.32:0/3313709706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 msgr2=0x55e0a7654a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7654a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 -- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a764a800 msgr2=0x55e0a764e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a764a800 0x55e0a764e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fde3d1640 1 -- 172.21.15.32:0/3313709706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a7383620 con 0x55e0a763f620 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.320+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7654a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:38.321 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009230 tx=0x7f4fd4004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db9331e5cb3cfa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4035040 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f4fd4004770 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e0a75b2390 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4fd4005610 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 10 client.16623 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e0a765d7a0 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e0a75b2390 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f4fd404d020 con 0x55e0a763f620 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 10 client.16623 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:38.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 20 client.16623 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a765ec10 0x55e0a7661040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 4 client.16623 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.321+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0a75ac200 con 0x55e0a765ec10 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.322+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a765ec10 0x55e0a7661040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.322+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a765ec10 0x55e0a7661040 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=130bc83b88dea8e3 server_cookie=a642e5f44b050c43 in_seq=0 out_seq=0 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.322+0000 7f4fc1ffb640 10 client.16623 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/3313709706 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0a75ac200 con 0x55e0a765ec10 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fc1ffb640 10 client.16623 handle_command_reply: tid=0 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 client.16623 shutdown 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 msgr2=0x55e0a76544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009230 tx=0x7f4fd4004580 comp rx=0 tx=0).stop 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 shutdown_connections 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a765ec10 0x55e0a7661040 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a764a800 0x55e0a764e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7654a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 --2- 172.21.15.32:0/3313709706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763f620 0x55e0a76544d0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fc1ffb640 0 client.16623 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 >> 172.21.15.32:0/3313709706 conn(0x55e0a763bb30 msgr2=0x55e0a7641860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 shutdown_connections 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 1 -- 172.21.15.32:0/3313709706 wait complete. 2024-04-23T18:24:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.323+0000 7f4fe342d740 20 client.16623 trim_cache size 0 max 16384 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.338+0000 7f4fe342d740 1 Processor -- start 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.338+0000 7f4fe342d740 1 -- start start 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a7654920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7646d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81ae110 con 0x55e0a764a800 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81ab9e0 con 0x55e0a763f620 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fe342d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81a9850 con 0x55e0a763ff40 2024-04-23T18:24:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44544/0 (socket says 172.21.15.32:44544) 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 -- 172.21.15.32:0/4285679364 learned_addr learned my addr 172.21.15.32:0/4285679364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7646d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a7654920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 -- 172.21.15.32:0/4285679364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 msgr2=0x55e0a7646d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7646d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 -- 172.21.15.32:0/4285679364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 msgr2=0x55e0a7654920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a7654920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 -- 172.21.15.32:0/4285679364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a764eba0 con 0x55e0a764a800 2024-04-23T18:24:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009230 tx=0x7f4fd0010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7db20a2472ef84 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.339+0000 7f4fc3fff640 1 -- 172.21.15.32:0/4285679364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0012550 con 0x55e0a764a800 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fc3fff640 1 -- 172.21.15.32:0/4285679364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0003e20 con 0x55e0a764a800 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fc3fff640 1 -- 172.21.15.32:0/4285679364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00050c0 con 0x55e0a764a800 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- 172.21.15.32:0/4285679364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 msgr2=0x55e0a7666950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009230 tx=0x7f4fd0010f90 comp rx=0 tx=0).stop 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- 172.21.15.32:0/4285679364 shutdown_connections 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a764a800 0x55e0a7666950 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a763ff40 0x55e0a7646d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 --2- 172.21.15.32:0/4285679364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a7654920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- 172.21.15.32:0/4285679364 >> 172.21.15.32:0/4285679364 conn(0x55e0a763bb30 msgr2=0x55e0a7646670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- 172.21.15.32:0/4285679364 shutdown_connections 2024-04-23T18:24:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- 172.21.15.32:0/4285679364 wait complete. 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.340+0000 7f4fe342d740 1 -- start start 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 Processor -- start 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a765af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a7658a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81a9ea0 con 0x55e0a763ff40 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81ab310 con 0x55e0a763f620 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81aa570 con 0x55e0a764a800 2024-04-23T18:24:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fe342d740 10 client.0 _ll_register_callbacks cb 0x7f4fd4004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44546/0 (socket says 172.21.15.32:44546) 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2847016256 learned_addr learned my addr 172.21.15.32:0/2847016256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a765af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a7658a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2847016256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 msgr2=0x55e0a7658a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a7658a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 msgr2=0x55e0a765af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a765af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fddbd0640 1 -- 172.21.15.32:0/2847016256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a76c41d0 con 0x55e0a763ff40 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.341+0000 7f4fdebd2640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a7658a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fddbd0640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80cdb40 tx=0x7f4fc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865862fd9036ab20 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc80a4fa0 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f4fc80989a0 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd0003e20 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4fc80cdd60 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 10 client.16629 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e0a76c5340 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0003e20 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f4fc80b5ab0 con 0x55e0a763ff40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 10 client.16629 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 20 client.16629 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a76c5aa0 0x55e0a76c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 4 client.16629 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0a75285f0 con 0x55e0a76c5aa0 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.342+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a76c5aa0 0x55e0a76c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.343+0000 7f4fc1ffb640 10 client.16629 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.343+0000 7f4fde3d1640 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a76c5aa0 0x55e0a76c7ec0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8fdbfe552ba0a6 server_cookie=3204fdb70346b29 in_seq=0 out_seq=0 2024-04-23T18:24:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.357+0000 7f4fc1ffb640 1 -- 172.21.15.32:0/2847016256 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0a75285f0 con 0x55e0a76c5aa0 2024-04-23T18:24:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.357+0000 7f4fc1ffb640 10 client.16629 handle_command_reply: tid=0 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.357+0000 7f4fe342d740 1 client.16629 shutdown 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 msgr2=0x55e0a7658310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80cdb40 tx=0x7f4fc8000bc0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 shutdown_connections 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e0a76c5aa0 0x55e0a76c7ec0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e0a764a800 0x55e0a7658a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e0a763ff40 0x55e0a7658310 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fc1ffb640 0 client.16629 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 --2- 172.21.15.32:0/2847016256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e0a763f620 0x55e0a765af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 >> 172.21.15.32:0/2847016256 conn(0x55e0a763bb30 msgr2=0x55e0a7646f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 shutdown_connections 2024-04-23T18:24:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 1 -- 172.21.15.32:0/2847016256 wait complete. 2024-04-23T18:24:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe342d740 20 client.16629 trim_cache size 0 max 16384 2024-04-23T18:24:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:24:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:24:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:24:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 msgr2=0x7f4fc40a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 0x7f4fc40a6370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0005c40 tx=0x7f4fd001d000 comp rx=0 tx=0).stop 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 msgr2=0x7f4fd8198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c970 tx=0x7f4fc800ce30 comp rx=0 tx=0).stop 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 shutdown_connections 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fd819d000 0x7f4fd819d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.358+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4fc40a3eb0 0x7f4fc40a6370 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.359+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd8107290 0x7f4fd8198cd0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.359+0000 7f4fe0635640 1 --2- 172.21.15.32:0/2655388233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fd81021c0 0x7f4fd8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.359+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 >> 172.21.15.32:0/2655388233 conn(0x7f4fd80fda40 msgr2=0x7f4fd80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.359+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 shutdown_connections 2024-04-23T18:24:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:38.359+0000 7f4fe0635640 1 -- 172.21.15.32:0/2655388233 wait complete. 2024-04-23T18:24:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 550.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:24:39.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:38 smithi040 ceph-mon[32626]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:39.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:38 smithi064 ceph-mon[33062]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:39.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:38 smithi032 ceph-mon[29221]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:40 smithi040 ceph-mon[32626]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:41.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:40 smithi064 ceph-mon[33062]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:41.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:40 smithi032 ceph-mon[29221]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 34 MiB/s wr, 1.78k op/s 2024-04-23T18:24:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:42 smithi040 ceph-mon[32626]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 260 op/s 2024-04-23T18:24:43.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:42 smithi064 ceph-mon[33062]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 260 op/s 2024-04-23T18:24:43.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:42 smithi032 ceph-mon[29221]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 260 op/s 2024-04-23T18:24:43.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 555.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:24:45.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:44 smithi040 ceph-mon[32626]: pgmap v298: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.8 MiB/s wr, 295 op/s 2024-04-23T18:24:45.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:44 smithi064 ceph-mon[33062]: pgmap v298: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.8 MiB/s wr, 295 op/s 2024-04-23T18:24:45.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:44 smithi032 ceph-mon[29221]: pgmap v298: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.8 MiB/s wr, 295 op/s 2024-04-23T18:24:47.059 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:46 smithi032 ceph-mon[29221]: pgmap v299: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.4 MiB/s wr, 352 op/s 2024-04-23T18:24:47.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:46 smithi040 ceph-mon[32626]: pgmap v299: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.4 MiB/s wr, 352 op/s 2024-04-23T18:24:47.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:46 smithi064 ceph-mon[33062]: pgmap v299: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.4 MiB/s wr, 352 op/s 2024-04-23T18:24:48.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 560.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:24:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:48 smithi040 ceph-mon[32626]: pgmap v300: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-23T18:24:49.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:48 smithi064 ceph-mon[33062]: pgmap v300: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-23T18:24:49.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:48 smithi032 ceph-mon[29221]: pgmap v300: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-23T18:24:50.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 -- 172.21.15.32:0/3765769369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 msgr2=0x7f204c101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 --2- 172.21.15.32:0/3765769369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c101120 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2048009900 tx=0x7f204802a410 comp rx=0 tx=0).stop 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 -- 172.21.15.32:0/3765769369 shutdown_connections 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 --2- 172.21.15.32:0/3765769369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 0x7f204c110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 --2- 172.21.15.32:0/3765769369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 --2- 172.21.15.32:0/3765769369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c101120 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.845+0000 7f20540d0640 1 -- 172.21.15.32:0/3765769369 >> 172.21.15.32:0/3765769369 conn(0x7f204c077f50 msgr2=0x7f204c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- 172.21.15.32:0/3765769369 shutdown_connections 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- 172.21.15.32:0/3765769369 wait complete. 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 Processor -- start 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- start start 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 0x7f204c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1143f0 con 0x7f204c100d40 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f204c114270 con 0x7f204c10e8f0 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.846+0000 7f20540d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f204c114570 con 0x7f204c101660 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55768/0 (socket says 172.21.15.32:55768) 2024-04-23T18:24:50.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 -- 172.21.15.32:0/613636620 learned_addr learned my addr 172.21.15.32:0/613636620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f205266d640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 0x7f204c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f205166b640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 -- 172.21.15.32:0/613636620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 msgr2=0x7f204c1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 -- 172.21.15.32:0/613636620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 msgr2=0x7f204c19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 0x7f204c19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 -- 172.21.15.32:0/613636620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c19afa0 con 0x7f204c100d40 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f205166b640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2051e6c640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2048005940 tx=0x7f20480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204803f030 con 0x7f204c100d40 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c19b230 con 0x7f204c100d40 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2048003cb0 con 0x7f204c100d40 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048035070 con 0x7f204c100d40 2024-04-23T18:24:50.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.847+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204c1a7460 con 0x7f204c100d40 2024-04-23T18:24:50.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.849+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204c06ae80 con 0x7f204c100d40 2024-04-23T18:24:50.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.850+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2048042030 con 0x7f204c100d40 2024-04-23T18:24:50.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.851+0000 7f2042ffd640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 0x7f20380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:50.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.851+0000 7f205166b640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 0x7f20380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:50.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.851+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f20480f1340 con 0x7f204c100d40 2024-04-23T18:24:50.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.851+0000 7f205166b640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 0x7f20380a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f204c1008e0 tx=0x7f203c00b5f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:50.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:50.855+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20480bda00 con 0x7f204c100d40 2024-04-23T18:24:51.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:50 smithi032 ceph-mon[29221]: pgmap v301: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-23T18:24:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.082+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f204c109a80 con 0x7f204c100d40 2024-04-23T18:24:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.082+0000 7f2042ffd640 1 -- 172.21.15.32:0/613636620 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11851 (secure 0 0 0) 0x7f20480bd300 con 0x7f204c100d40 2024-04-23T18:24:51.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:24:51.076 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:24:30.869819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:24:51.076 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-04-23T18:24:51.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 msgr2=0x7f20380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 0x7f20380a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f204c1008e0 tx=0x7f203c00b5f0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 msgr2=0x7f204c1a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2048005940 tx=0x7f20480042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 shutdown_connections 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c10e8f0 0x7f204c19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f204c101660 0x7f204c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f20380a3d20 0x7f20380a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 --2- 172.21.15.32:0/613636620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204c100d40 0x7f204c1a0610 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 >> 172.21.15.32:0/613636620 conn(0x7f204c077f50 msgr2=0x7f204c10ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 shutdown_connections 2024-04-23T18:24:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.084+0000 7f20540d0640 1 -- 172.21.15.32:0/613636620 wait complete. 2024-04-23T18:24:51.089 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id d0181b4e --release --await' 2024-04-23T18:24:51.089 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id d0181b4e --release --await 2024-04-23T18:24:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:50 smithi040 ceph-mon[32626]: pgmap v301: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-23T18:24:51.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:50 smithi064 ceph-mon[33062]: pgmap v301: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 -- 172.21.15.32:0/3994846492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e8100fb0 msgr2=0x7f00e8101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 --2- 172.21.15.32:0/3994846492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e8100fb0 0x7f00e8101390 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009900 tx=0x7f00e402a480 comp rx=0 tx=0).stop 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 -- 172.21.15.32:0/3994846492 shutdown_connections 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 --2- 172.21.15.32:0/3994846492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e81062d0 0x7f00e8106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 --2- 172.21.15.32:0/3994846492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e8105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 --2- 172.21.15.32:0/3994846492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e8100fb0 0x7f00e8101390 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 -- 172.21.15.32:0/3994846492 >> 172.21.15.32:0/3994846492 conn(0x7f00e8078000 msgr2=0x7f00e8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 -- 172.21.15.32:0/3994846492 shutdown_connections 2024-04-23T18:24:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.205+0000 7f00efdad640 1 -- 172.21.15.32:0/3994846492 wait complete. 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 Processor -- start 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 -- start start 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 0x7f00e819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81187a0 con 0x7f00e81062d0 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8118620 con 0x7f00e8100fb0 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00efdad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8118920 con 0x7f00e81018d0 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00ee34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00ee34a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55796/0 (socket says 172.21.15.32:55796) 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00ee34a640 1 -- 172.21.15.32:0/946057307 learned_addr learned my addr 172.21.15.32:0/946057307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.206+0000 7f00ed348640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00edb49640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 0x7f00e819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 -- 172.21.15.32:0/946057307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 msgr2=0x7f00e819f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e819f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 -- 172.21.15.32:0/946057307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 msgr2=0x7f00e819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 0x7f00e819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 -- 172.21.15.32:0/946057307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e81a3d30 con 0x7f00e81062d0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ed348640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00edb49640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 0x7f00e819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00ee34a640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e960 tx=0x7f00e000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00deffd640 1 -- 172.21.15.32:0/946057307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0004d20 con 0x7f00e81062d0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e81a4020 con 0x7f00e81062d0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00deffd640 1 -- 172.21.15.32:0/946057307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e0005020 con 0x7f00e81062d0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00deffd640 1 -- 172.21.15.32:0/946057307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e00116d0 con 0x7f00e81062d0 2024-04-23T18:24:51.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.207+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e81ab8b0 con 0x7f00e81062d0 2024-04-23T18:24:51.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.209+0000 7f00deffd640 1 -- 172.21.15.32:0/946057307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f00e00051d0 con 0x7f00e81062d0 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 Processor -- start 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 -- start start 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e81540f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81541480 0x555e81543870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e8149b6d0 con 0x555e81536490 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555e814381a0 con 0x555e81541480 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.210+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555e81497690 con 0x555e81536db0 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00deffd640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 0x7f00d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55800/0 (socket says 172.21.15.32:55800) 2024-04-23T18:24:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 -- 172.21.15.32:0/3720346789 learned_addr learned my addr 172.21.15.32:0/3720346789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00edb49640 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e81540f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81541480 0x555e81543870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ee34a640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 0x7f00d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 -- 172.21.15.32:0/3720346789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 msgr2=0x555e81540f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e81540f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00deffd640 1 -- 172.21.15.32:0/946057307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f00e00d0970 con 0x7f00e81062d0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 -- 172.21.15.32:0/3720346789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81541480 msgr2=0x555e81543870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81541480 0x555e81543870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 -- 172.21.15.32:0/3720346789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e814425b0 con 0x555e81536490 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ed348640 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f00d8013660 tx=0x7f00d800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2549c25589b193 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00ee34a640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 0x7f00d40a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f00e00ce190 tx=0x7f00e00bd040 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00d3fff640 1 -- 172.21.15.32:0/3720346789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8019030 con 0x555e81536490 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00d3fff640 1 -- 172.21.15.32:0/3720346789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d80054d0 con 0x555e81536490 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.211+0000 7f00d3fff640 1 -- 172.21.15.32:0/3720346789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8011540 con 0x555e81536490 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3720346789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 msgr2=0x555e81536870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f00d8013660 tx=0x7f00d800b760 comp rx=0 tx=0).stop 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3720346789 shutdown_connections 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81541480 0x555e81543870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e81540f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3720346789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e81536870 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3720346789 >> 172.21.15.32:0/3720346789 conn(0x555e815327b0 msgr2=0x555e81532b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3720346789 shutdown_connections 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3720346789 wait complete. 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.212+0000 7f00f2ba0740 1 -- start start 2024-04-23T18:24:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 Processor -- start 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81540f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e8153e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e81544310 con 0x555e81541480 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555e81435b20 con 0x555e81536490 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555e81544490 con 0x555e81536db0 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00f2ba0740 10 client.0 _ll_register_callbacks cb 0x555e81435530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55802/0 (socket says 172.21.15.32:55802) 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 -- 172.21.15.32:0/3486545108 learned_addr learned my addr 172.21.15.32:0/3486545108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81540f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00edb49640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e8153e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 -- 172.21.15.32:0/3486545108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 msgr2=0x555e8153e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e8153e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 -- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 msgr2=0x555e81540f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81540f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.213+0000 7f00ed348640 1 -- 172.21.15.32:0/3486545108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e811ad920 con 0x555e81541480 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00ed348640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00d8013630 tx=0x7f00d8004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1940d08a77738317 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8019040 con 0x555e81541480 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00d80054d0 con 0x555e81541480 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f00d80051b0 con 0x555e81541480 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f00d80046c0 con 0x555e81541480 2024-04-23T18:24:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 10 client.16647 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555e81279d30 con 0x555e81541480 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d80054d0 con 0x555e81541480 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f00d802a020 con 0x555e81541480 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 10 client.16647 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 10 client.16647 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 20 client.16647 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e81551640 0x555e81553a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 4 client.16647 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.214+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e81459500 con 0x555e81551640 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.215+0000 7f00edb49640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e81551640 0x555e81553a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.215+0000 7f00edb49640 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e81551640 0x555e81553a30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6f8f30a6857fd1f server_cookie=43fe48b046b45f37 in_seq=0 out_seq=0 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.215+0000 7f00d17fa640 10 client.16647 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00d17fa640 1 -- 172.21.15.32:0/3486545108 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e81459500 con 0x555e81551640 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00d17fa640 10 client.16647 handle_command_reply: tid=0 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 client.16647 shutdown 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 msgr2=0x555e8153e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00d8013630 tx=0x7f00d8004cd0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 shutdown_connections 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e81551640 0x555e81553a30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81541480 0x555e8153e8f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00d17fa640 0 client.16647 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81536db0 0x555e8153e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3486545108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81540f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 >> 172.21.15.32:0/3486545108 conn(0x555e815327b0 msgr2=0x555e815384b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 shutdown_connections 2024-04-23T18:24:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3486545108 wait complete. 2024-04-23T18:24:51.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.216+0000 7f00f2ba0740 20 client.16647 trim_cache size 0 max 16384 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 Processor -- start 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 -- start start 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 0x555e81540a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81559120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b6a60 con 0x555e81536490 2024-04-23T18:24:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b4330 con 0x555e81536db0 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b21a0 con 0x555e81541480 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55806/0 (socket says 172.21.15.32:55806) 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 -- 172.21.15.32:0/3410572906 learned_addr learned my addr 172.21.15.32:0/3410572906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00edb49640 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 0x555e81540a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 -- 172.21.15.32:0/3410572906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 msgr2=0x555e81559120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81559120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 -- 172.21.15.32:0/3410572906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 msgr2=0x555e81540a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 0x555e81540a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 -- 172.21.15.32:0/3410572906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e81536be0 con 0x555e81536490 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00edb49640 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 0x555e81540a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00ed348640 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f00d8004ca0 tx=0x7f00d800bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651f88f25e331bec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.234+0000 7f00d37fe640 1 -- 172.21.15.32:0/3410572906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8019040 con 0x555e81536490 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00d37fe640 1 -- 172.21.15.32:0/3410572906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d8004d40 con 0x555e81536490 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00d37fe640 1 -- 172.21.15.32:0/3410572906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d80050d0 con 0x555e81536490 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3410572906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 msgr2=0x555e8154d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f00d8004ca0 tx=0x7f00d800bbf0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3410572906 shutdown_connections 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81559120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536db0 0x555e81540a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3410572906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536490 0x555e8154d230 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3410572906 >> 172.21.15.32:0/3410572906 conn(0x555e815327b0 msgr2=0x555e81537be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3410572906 shutdown_connections 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3410572906 wait complete. 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.235+0000 7f00f2ba0740 1 -- start start 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 Processor -- start 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81542860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81547d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b27f0 con 0x555e81536db0 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b3c60 con 0x555e81536490 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b2ec0 con 0x555e81541480 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00f2ba0740 10 client.0 _ll_register_callbacks cb 0x7f00d8005020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55812/0 (socket says 172.21.15.32:55812) 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 -- 172.21.15.32:0/3732268341 learned_addr learned my addr 172.21.15.32:0/3732268341 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00ed348640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81547d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00ed348640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81542860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 -- 172.21.15.32:0/3732268341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 msgr2=0x555e81547d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81547d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 -- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 msgr2=0x555e81542860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81542860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 -- 172.21.15.32:0/3732268341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e815482c0 con 0x555e81536db0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00ed348640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81542860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00edb49640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f00e4031440 tx=0x7f00e40041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947d811725587981 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.236+0000 7f00d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4035040 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f00e4044050 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00d8004d40 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f00e4051050 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 10 client.16653 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555e815b80a0 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d8004d40 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f00e40446e0 con 0x555e81536db0 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 10 client.16653 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 10 client.16653 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 20 client.16653 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e815b8870 0x555e815bac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 4 client.16653 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d0181b4e", "release": true, "await": true}] 2024-04-23T18:24:51.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.237+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d0181b4e", "release": true, "await": true}) -- 0x555e814a90f0 con 0x555e815b8870 2024-04-23T18:24:51.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.238+0000 7f00ed348640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e815b8870 0x555e815bac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.238+0000 7f00ed348640 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e815b8870 0x555e815bac90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caef9f95c7f1025d server_cookie=26017ec3c04d594b in_seq=0 out_seq=0 2024-04-23T18:24:51.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.238+0000 7f00d17fa640 10 client.16653 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.251+0000 7f00d17fa640 1 -- 172.21.15.32:0/3732268341 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555e814a90f0 con 0x555e815b8870 2024-04-23T18:24:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.251+0000 7f00d17fa640 10 client.16653 handle_command_reply: tid=0 2024-04-23T18:24:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.251+0000 7f00f2ba0740 1 client.16653 shutdown 2024-04-23T18:24:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 msgr2=0x555e81542da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f00e4031440 tx=0x7f00e40041a0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 shutdown_connections 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555e815b8870 0x555e815bac90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e81541480 0x555e81547d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e81536db0 0x555e81542da0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00d17fa640 0 client.16653 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 --2- 172.21.15.32:0/3732268341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e81536490 0x555e81542860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 >> 172.21.15.32:0/3732268341 conn(0x555e815327b0 msgr2=0x555e81538470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 shutdown_connections 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 1 -- 172.21.15.32:0/3732268341 wait complete. 2024-04-23T18:24:51.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00f2ba0740 20 client.16653 trim_cache size 0 max 16384 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 40, 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 32, 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "d0181b4e": { 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 32, 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:24:51.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 77.7, 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:51.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:24:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:24:51.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 msgr2=0x7f00d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.252+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 0x7f00d40a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f00e00ce190 tx=0x7f00e00bd040 comp rx=0 tx=0).stop 2024-04-23T18:24:51.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 msgr2=0x7f00e81a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e960 tx=0x7f00e000ee20 comp rx=0 tx=0).stop 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 shutdown_connections 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f00d40a3d20 0x7f00d40a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e81062d0 0x7f00e81a3600 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e81018d0 0x7f00e819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 --2- 172.21.15.32:0/946057307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00e8100fb0 0x7f00e819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 >> 172.21.15.32:0/946057307 conn(0x7f00e8078000 msgr2=0x7f00e8103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 shutdown_connections 2024-04-23T18:24:51.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:24:51.253+0000 7f00efdad640 1 -- 172.21.15.32:0/946057307 wait complete. 2024-04-23T18:24:51.262 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd0181b4e', seconds elapsed: 20.0 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 -- 172.21.15.32:0/351709388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773410e660 msgr2=0x7f773410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 --2- 172.21.15.32:0/351709388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773410e660 0x7f773410ea40 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7724009860 tx=0x7f772402a340 comp rx=0 tx=0).stop 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 -- 172.21.15.32:0/351709388 shutdown_connections 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 --2- 172.21.15.32:0/351709388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f7734115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 --2- 172.21.15.32:0/351709388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77341085a0 0x7f7734108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 --2- 172.21.15.32:0/351709388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773410e660 0x7f773410ea40 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 -- 172.21.15.32:0/351709388 >> 172.21.15.32:0/351709388 conn(0x7f7734076500 msgr2=0x7f7734078920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 -- 172.21.15.32:0/351709388 shutdown_connections 2024-04-23T18:24:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.482+0000 7f7739d44640 1 -- 172.21.15.32:0/351709388 wait complete. 2024-04-23T18:24:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 Processor -- start 2024-04-23T18:24:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 -- start start 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77341085a0 0x7f77341a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 0x7f773419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7734118810 con 0x7f77341085a0 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7734118690 con 0x7f773410e660 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.483+0000 7f7739d44640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7734118990 con 0x7f7734108f60 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:38724/0 (socket says 172.21.15.32:38724) 2024-04-23T18:24:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7733fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 0x7f773419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 -- 172.21.15.32:0/1847444631 learned_addr learned my addr 172.21.15.32:0/1847444631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 -- 172.21.15.32:0/1847444631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 msgr2=0x7f773419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 0x7f773419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 -- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77341085a0 msgr2=0x7f77341a4a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77341085a0 0x7f77341a4a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7732ffd640 1 -- 172.21.15.32:0/1847444631 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773419f3d0 con 0x7f7734108f60 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.484+0000 7f7733fff640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 0x7f773419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7732ffd640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f771c00e9c0 tx=0x7f771c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7730ff9640 1 -- 172.21.15.32:0/1847444631 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c00cde0 con 0x7f7734108f60 2024-04-23T18:24:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7730ff9640 1 -- 172.21.15.32:0/1847444631 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f771c0035f0 con 0x7f7734108f60 2024-04-23T18:24:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7730ff9640 1 -- 172.21.15.32:0/1847444631 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c010690 con 0x7f7734108f60 2024-04-23T18:24:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773419f660 con 0x7f7734108f60 2024-04-23T18:24:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.485+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f773407a7a0 con 0x7f7734108f60 2024-04-23T18:24:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.488+0000 7f7730ff9640 1 -- 172.21.15.32:0/1847444631 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f771c004000 con 0x7f7734108f60 2024-04-23T18:24:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.489+0000 7f7730ff9640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 0x7f77200a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.489+0000 7f7730ff9640 1 -- 172.21.15.32:0/1847444631 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f771c0d0bd0 con 0x7f7734108f60 2024-04-23T18:24:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.489+0000 7f77337fe640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 0x7f77200a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.489+0000 7f773ca69740 1 Processor -- start 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.490+0000 7f773ca69740 1 -- start start 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f365e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 0x560f365f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f77337fe640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 0x7f77200a6250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f772402a860 tx=0x7f7724000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f3654c970 con 0x560f365e7f40 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55830/0 (socket says 172.21.15.32:55830) 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f7732ffd640 1 -- 172.21.15.32:0/4210406661 learned_addr learned my addr 172.21.15.32:0/4210406661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f364e94a0 con 0x560f365f2800 2024-04-23T18:24:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f77337fe640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f365e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f7733fff640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 0x560f365f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.491+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560f36548930 con 0x560f365e7620 2024-04-23T18:24:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f77337fe640 1 -- 172.21.15.32:0/4210406661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 msgr2=0x560f365f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f77337fe640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 0x560f365f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f77337fe640 1 -- 172.21.15.32:0/4210406661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 msgr2=0x560f365f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f77337fe640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f77337fe640 1 -- 172.21.15.32:0/4210406661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f3625f210 con 0x560f365e7620 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f7732ffd640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.492+0000 7f7733fff640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 0x560f365f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.493+0000 7f77337fe640 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f365e7a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7724035040 tx=0x7f7724039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b16ec929faf96755 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.493+0000 7f7729ffb640 1 -- 172.21.15.32:0/4210406661 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724037040 con 0x560f365e7620 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.493+0000 7f7729ffb640 1 -- 172.21.15.32:0/4210406661 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7724004040 con 0x560f365e7620 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 msgr2=0x560f365e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f365e7a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7724035040 tx=0x7f7724039000 comp rx=0 tx=0).stop 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 shutdown_connections 2024-04-23T18:24:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365f2800 0x560f365f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 --2- 172.21.15.32:0/4210406661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f365e7a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.494+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 >> 172.21.15.32:0/4210406661 conn(0x560f365e3b30 msgr2=0x560f365e3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.497+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 shutdown_connections 2024-04-23T18:24:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.497+0000 7f773ca69740 1 -- 172.21.15.32:0/4210406661 wait complete. 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.498+0000 7f773ca69740 1 -- start start 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.498+0000 7f773ca69740 1 Processor -- start 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7620 0x560f365f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f365f5370 con 0x560f365e7f40 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f364e70b0 con 0x560f365e7620 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560f365f54f0 con 0x560f365f2800 2024-04-23T18:24:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.500+0000 7f773ca69740 10 client.0 _ll_register_callbacks cb 0x560f364d4d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.501+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.501+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55844/0 (socket says 172.21.15.32:55844) 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.501+0000 7f7732ffd640 1 -- 172.21.15.32:0/789805372 learned_addr learned my addr 172.21.15.32:0/789805372 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.501+0000 7f77337fe640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7620 0x560f365f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 -- 172.21.15.32:0/789805372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 msgr2=0x560f365ef8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ef8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 -- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7620 msgr2=0x560f365f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7620 0x560f365f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 -- 172.21.15.32:0/789805372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f3632b620 con 0x560f365e7f40 2024-04-23T18:24:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.502+0000 7f7732ffd640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f771c09c890 tx=0x7f771c09d3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cec50f909c2ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.503+0000 7f7712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:24:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.504+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c0b5020 con 0x560f365e7f40 2024-04-23T18:24:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.504+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7724004040 con 0x560f365e7f40 2024-04-23T18:24:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.505+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f771c003aa0 con 0x560f365e7f40 2024-04-23T18:24:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.505+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f771c080050 con 0x560f365e7f40 2024-04-23T18:24:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.507+0000 7f773ca69740 10 client.16662 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.507+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560f365f5980 con 0x560f365e7f40 2024-04-23T18:24:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.514+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7724004040 con 0x560f365e7f40 2024-04-23T18:24:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.514+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f771c099200 con 0x560f365e7f40 2024-04-23T18:24:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 10 client.16662 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 10 client.16662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 20 client.16662 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f365f6e40 0x560f365f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 4 client.16662 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.516+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f3655a390 con 0x560f365f6e40 2024-04-23T18:24:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.518+0000 7f77337fe640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f365f6e40 0x560f365f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.524+0000 7f77337fe640 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f365f6e40 0x560f365f9270 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c0722dd60e81cfd server_cookie=a39fa3147c9b5674 in_seq=0 out_seq=0 2024-04-23T18:24:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.524+0000 7f7712ffd640 10 client.16662 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.526+0000 7f7712ffd640 1 -- 172.21.15.32:0/789805372 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f3655a390 con 0x560f365f6e40 2024-04-23T18:24:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.526+0000 7f7712ffd640 10 client.16662 handle_command_reply: tid=0 2024-04-23T18:24:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.526+0000 7f773ca69740 1 client.16662 shutdown 2024-04-23T18:24:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.527+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 msgr2=0x560f365ef3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.527+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f771c09c890 tx=0x7f771c09d3d0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 shutdown_connections 2024-04-23T18:24:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f365f6e40 0x560f365f9270 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7f40 0x560f365ef3b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 --2- 172.21.15.32:0/789805372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7620 0x560f365f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 >> 172.21.15.32:0/789805372 conn(0x560f365e3b30 msgr2=0x560f365e98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f7712ffd640 0 client.16662 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 shutdown_connections 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 1 -- 172.21.15.32:0/789805372 wait complete. 2024-04-23T18:24:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.528+0000 7f773ca69740 20 client.16662 trim_cache size 0 max 16384 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.551+0000 7f773ca69740 1 Processor -- start 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.551+0000 7f773ca69740 1 -- start start 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f365ff050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ec610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f773406e410 con 0x560f365e7620 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f773406bce0 con 0x560f365e7f40 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f773ca69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7734081190 con 0x560f365f2800 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55848/0 (socket says 172.21.15.32:55848) 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 -- 172.21.15.32:0/150790793 learned_addr learned my addr 172.21.15.32:0/150790793 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f7732ffd640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f365ff050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f7733fff640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ec610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 -- 172.21.15.32:0/150790793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 msgr2=0x560f365ec610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ec610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 -- 172.21.15.32:0/150790793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 msgr2=0x560f365ff050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f365ff050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f77337fe640 1 -- 172.21.15.32:0/150790793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f365df2e0 con 0x560f365e7620 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.552+0000 7f7732ffd640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f365ff050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.553+0000 7f77337fe640 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f772402a930 tx=0x7f772403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1fa207dd10a36a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.553+0000 7f77297fa640 1 -- 172.21.15.32:0/150790793 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724058070 con 0x560f365e7620 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.553+0000 7f77297fa640 1 -- 172.21.15.32:0/150790793 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7724003b30 con 0x560f365e7620 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.553+0000 7f77297fa640 1 -- 172.21.15.32:0/150790793 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724037040 con 0x560f365e7620 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- 172.21.15.32:0/150790793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 msgr2=0x560f365feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f772402a930 tx=0x7f772403c000 comp rx=0 tx=0).stop 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- 172.21.15.32:0/150790793 shutdown_connections 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365f2800 0x560f365ec610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f365ff050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 --2- 172.21.15.32:0/150790793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365e7620 0x560f365feb10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- 172.21.15.32:0/150790793 >> 172.21.15.32:0/150790793 conn(0x560f365e3b30 msgr2=0x560f365ef4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- 172.21.15.32:0/150790793 shutdown_connections 2024-04-23T18:24:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- 172.21.15.32:0/150790793 wait complete. 2024-04-23T18:24:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.554+0000 7f773ca69740 1 -- start start 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 Processor -- start 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f36607e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365f2800 0x560f366088e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77340817e0 con 0x560f365f2800 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7734082c50 con 0x560f365e7f40 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7734081eb0 con 0x560f365e7620 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f773ca69740 10 client.0 _ll_register_callbacks cb 0x7f771c09c510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f7732ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:58266/0 (socket says 172.21.15.32:58266) 2024-04-23T18:24:51.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f7732ffd640 1 -- 172.21.15.32:0/370991427 learned_addr learned my addr 172.21.15.32:0/370991427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.555+0000 7f7733fff640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365f2800 0x560f366088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.556+0000 7f7732ffd640 1 -- 172.21.15.32:0/370991427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 msgr2=0x560f36607e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.556+0000 7f7732ffd640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f36607e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.556+0000 7f7732ffd640 1 -- 172.21.15.32:0/370991427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365f2800 msgr2=0x560f366088e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.556+0000 7f7732ffd640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365f2800 0x560f366088e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.556+0000 7f7732ffd640 1 -- 172.21.15.32:0/370991427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f36689700 con 0x560f365e7f40 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f7732ffd640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f771c09cd50 tx=0x7f771c0b6220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c944fb407e706d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f7712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:24:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c0b5040 con 0x560f365e7f40 2024-04-23T18:24:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7724003b30 con 0x560f365e7f40 2024-04-23T18:24:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7f771c0042f0 con 0x560f365e7f40 2024-04-23T18:24:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.557+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f771c0bd0b0 con 0x560f365e7f40 2024-04-23T18:24:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 10 client.26222 fetch_fsmap learned FSMap version 40 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560f3668aa10 con 0x560f365e7f40 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7724003b30 con 0x560f365e7f40 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f771c09ba50 con 0x560f365e7f40 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 10 client.26222 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 10 client.26222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 20 client.26222 populate_metadata read hostname 'smithi032' 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f3668b110 0x560f3668d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 4 client.26222 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.558+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f36554200 con 0x560f3668b110 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.559+0000 7f77337fe640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f3668b110 0x560f3668d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.559+0000 7f7712ffd640 10 client.26222 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.559+0000 7f77337fe640 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f3668b110 0x560f3668d530 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=772512ba695025d server_cookie=f60bbb3fa0396d5f in_seq=0 out_seq=0 2024-04-23T18:24:51.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.582+0000 7f7712ffd640 1 -- 172.21.15.32:0/370991427 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f36554200 con 0x560f3668b110 2024-04-23T18:24:51.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f7712ffd640 10 client.26222 handle_command_reply: tid=0 2024-04-23T18:24:51.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 client.26222 shutdown 2024-04-23T18:24:51.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 msgr2=0x560f36608360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f771c09cd50 tx=0x7f771c0b6220 comp rx=0 tx=0).stop 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 shutdown_connections 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560f3668b110 0x560f3668d530 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f365f2800 0x560f366088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f7712ffd640 0 client.26222 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f365e7f40 0x560f36608360 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 --2- 172.21.15.32:0/370991427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560f365e7620 0x560f36607e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 >> 172.21.15.32:0/370991427 conn(0x560f365e3b30 msgr2=0x560f365e2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.583+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 shutdown_connections 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f773ca69740 1 -- 172.21.15.32:0/370991427 wait complete. 2024-04-23T18:24:51.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f773ca69740 20 client.26222 trim_cache size 0 max 16384 2024-04-23T18:24:51.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:24:51.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:24:51.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:24:51.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 msgr2=0x7f77200a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 0x7f77200a6250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f772402a860 tx=0x7f7724000bc0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 msgr2=0x7f77341a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.584+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f771c00e9c0 tx=0x7f771c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 shutdown_connections 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f773410e660 0x7f773419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7734108f60 0x7f77341a4f90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f77200a3d90 0x7f77200a6250 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 --2- 172.21.15.32:0/1847444631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77341085a0 0x7f77341a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 >> 172.21.15.32:0/1847444631 conn(0x7f7734076500 msgr2=0x7f7734077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 shutdown_connections 2024-04-23T18:24:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:24:51.585+0000 7f7739d44640 1 -- 172.21.15.32:0/1847444631 wait complete. 2024-04-23T18:24:52.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:51 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/613636620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:52.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:51 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/613636620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:52.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:51 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/613636620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:52 smithi040 ceph-mon[32626]: pgmap v302: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-23T18:24:53.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:52 smithi064 ceph-mon[33062]: pgmap v302: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-23T18:24:53.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:52 smithi032 ceph-mon[29221]: pgmap v302: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-23T18:24:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 565.0 s, 1538.5 tps, lat 273.943 ms stddev 2093.353 2024-04-23T18:24:55.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:54 smithi040 ceph-mon[32626]: pgmap v303: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-23T18:24:55.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:54 smithi064 ceph-mon[33062]: pgmap v303: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-23T18:24:55.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:54 smithi032 ceph-mon[29221]: pgmap v303: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-23T18:24:57.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:56 smithi040 ceph-mon[32626]: pgmap v304: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.0 MiB/s wr, 267 op/s 2024-04-23T18:24:57.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:56 smithi064 ceph-mon[33062]: pgmap v304: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.0 MiB/s wr, 267 op/s 2024-04-23T18:24:57.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:56 smithi032 ceph-mon[29221]: pgmap v304: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.0 MiB/s wr, 267 op/s 2024-04-23T18:24:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 570.0 s, 2717.6 tps, lat 36.758 ms stddev 40.226 2024-04-23T18:24:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:24:58 smithi040 ceph-mon[32626]: pgmap v305: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-04-23T18:24:59.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:24:58 smithi064 ceph-mon[33062]: pgmap v305: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-04-23T18:24:59.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:24:58 smithi032 ceph-mon[29221]: pgmap v305: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-04-23T18:25:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:00 smithi040 ceph-mon[32626]: pgmap v306: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 12 MiB/s wr, 489 op/s 2024-04-23T18:25:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:00 smithi064 ceph-mon[33062]: pgmap v306: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 12 MiB/s wr, 489 op/s 2024-04-23T18:25:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:00 smithi032 ceph-mon[29221]: pgmap v306: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 12 MiB/s wr, 489 op/s 2024-04-23T18:25:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:02 smithi040 ceph-mon[32626]: pgmap v307: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 484 op/s 2024-04-23T18:25:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:02 smithi064 ceph-mon[33062]: pgmap v307: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 484 op/s 2024-04-23T18:25:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:02 smithi032 ceph-mon[29221]: pgmap v307: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 484 op/s 2024-04-23T18:25:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 575.0 s, 1373.6 tps, lat 70.585 ms stddev 131.706 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.720+0000 7fda45b5d640 1 -- 172.21.15.32:0/3969918604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 msgr2=0x7fda40102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.720+0000 7fda45b5d640 1 --2- 172.21.15.32:0/3969918604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda40102580 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fda38009ca0 tx=0x7fda3802a8e0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 -- 172.21.15.32:0/3969918604 shutdown_connections 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 --2- 172.21.15.32:0/3969918604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 --2- 172.21.15.32:0/3969918604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40102b50 0x7fda40106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 --2- 172.21.15.32:0/3969918604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda40102580 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 -- 172.21.15.32:0/3969918604 >> 172.21.15.32:0/3969918604 conn(0x7fda400fda20 msgr2=0x7fda400ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 -- 172.21.15.32:0/3969918604 shutdown_connections 2024-04-23T18:25:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.721+0000 7fda45b5d640 1 -- 172.21.15.32:0/3969918604 wait complete. 2024-04-23T18:25:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.722+0000 7fda45b5d640 1 Processor -- start 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.722+0000 7fda45b5d640 1 -- start start 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda4019e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 0x7fda40198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda40112280 con 0x7fda401021a0 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda40112100 con 0x7fda40107270 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda45b5d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fda40112400 con 0x7fda40198660 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 0x7fda40198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda3effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda3effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4019e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38194/0 (socket says 172.21.15.32:38194) 2024-04-23T18:25:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.723+0000 7fda3effd640 1 -- 172.21.15.32:0/2734147098 learned_addr learned my addr 172.21.15.32:0/2734147098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda4019e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 -- 172.21.15.32:0/2734147098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 msgr2=0x7fda4019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 -- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 msgr2=0x7fda4019e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda4019e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 -- 172.21.15.32:0/2734147098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda40199180 con 0x7fda40198660 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.724+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 0x7fda40198b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda3400e970 tx=0x7fda3400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.725+0000 7fda3cff9640 1 -- 172.21.15.32:0/2734147098 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3400ccc0 con 0x7fda40198660 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.725+0000 7fda3cff9640 1 -- 172.21.15.32:0/2734147098 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda34005020 con 0x7fda40198660 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.725+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda40199470 con 0x7fda40198660 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.725+0000 7fda3cff9640 1 -- 172.21.15.32:0/2734147098 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34004a00 con 0x7fda40198660 2024-04-23T18:25:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.725+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda401a5620 con 0x7fda40198660 2024-04-23T18:25:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.728+0000 7fda48876740 1 Processor -- start 2024-04-23T18:25:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.728+0000 7fda3cff9640 1 -- 172.21.15.32:0/2734147098 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fda34004ba0 con 0x7fda40198660 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 -- start start 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda3cff9640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 0x7fda280a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda3cff9640 1 -- 172.21.15.32:0/2734147098 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7fda340d0780 con 0x7fda40198660 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5d6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb53b970 con 0x55eaeb5e1800 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb4d84a0 con 0x55eaeb5d6620 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.729+0000 7fda48876740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb537930 con 0x55eaeb5d6f40 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3effd640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 0x7fda280a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48780/0 (socket says 172.21.15.32:48780) 2024-04-23T18:25:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 -- 172.21.15.32:0/2888633387 learned_addr learned my addr 172.21.15.32:0/2888633387 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5d6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 -- 172.21.15.32:0/2888633387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 msgr2=0x55eaeb5e12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 -- 172.21.15.32:0/2888633387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 msgr2=0x55eaeb5d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5d6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 -- 172.21.15.32:0/2888633387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaeb24e210 con 0x55eaeb5e1800 2024-04-23T18:25:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3effd640 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3effd640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 0x7fda280a61d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda40101140 tx=0x7fda300156d0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.730+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fda340a4e80 tx=0x7fda3409ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df222b817c274d4d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda25ffb640 1 -- 172.21.15.32:0/2888633387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3401c040 con 0x55eaeb5e1800 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda25ffb640 1 -- 172.21.15.32:0/2888633387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340a0af0 con 0x55eaeb5e1800 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda25ffb640 1 -- 172.21.15.32:0/2888633387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3401c040 con 0x55eaeb5e1800 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 -- 172.21.15.32:0/2888633387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 msgr2=0x55eaeb5e3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fda340a4e80 tx=0x7fda3409ce40 comp rx=0 tx=0).stop 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 -- 172.21.15.32:0/2888633387 shutdown_connections 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e3bf0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 --2- 172.21.15.32:0/2888633387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5d6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 -- 172.21.15.32:0/2888633387 >> 172.21.15.32:0/2888633387 conn(0x55eaeb5d2b30 msgr2=0x55eaeb5d2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 -- 172.21.15.32:0/2888633387 shutdown_connections 2024-04-23T18:25:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.731+0000 7fda48876740 1 -- 172.21.15.32:0/2888633387 wait complete. 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 -- start start 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 Processor -- start 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5de430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5de970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb5e4430 con 0x55eaeb5d6f40 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb4d60b0 con 0x55eaeb5d6620 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55eaeb5e45b0 con 0x55eaeb5e1800 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda48876740 10 client.0 _ll_register_callbacks cb 0x55eaeb4c3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda3effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5de430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38212/0 (socket says 172.21.15.32:38212) 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda3f7fe640 1 -- 172.21.15.32:0/3240532632 learned_addr learned my addr 172.21.15.32:0/3240532632 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.732+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 -- 172.21.15.32:0/3240532632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 msgr2=0x55eaeb5de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 -- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 msgr2=0x55eaeb5e1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 -- 172.21.15.32:0/3240532632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaeb31a620 con 0x55eaeb5d6f40 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda3effd640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5de430 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fda400fde20 tx=0x7fda30004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08740b7bd3a6466 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda1effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30029060 con 0x55eaeb5d6f40 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda340a0af0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7fda30017930 con 0x55eaeb5d6f40 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fda30017b30 con 0x55eaeb5d6f40 2024-04-23T18:25:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda48876740 10 client.16680 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.733+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55eaeb5f0390 con 0x55eaeb5d6f40 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340a0af0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fda300233a0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 10 client.16680 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 20 client.16680 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb5f1800 0x55eaeb5f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 4 client.16680 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eaeb549390 con 0x55eaeb5f1800 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.734+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb5f1800 0x55eaeb5f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.735+0000 7fda1effd640 10 client.16680 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.735+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb5f1800 0x55eaeb5f3bf0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40b3ce8db6ad665e server_cookie=2a00f6f7a2a8b86d in_seq=0 out_seq=0 2024-04-23T18:25:04.733 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.735+0000 7fda1effd640 1 -- 172.21.15.32:0/3240532632 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eaeb549390 con 0x55eaeb5f1800 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.735+0000 7fda1effd640 10 client.16680 handle_command_reply: tid=0 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 client.16680 shutdown 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 msgr2=0x55eaeb5de430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5de430 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fda400fde20 tx=0x7fda30004700 comp rx=0 tx=0).stop 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 shutdown_connections 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb5f1800 0x55eaeb5f3bf0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5de430 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 --2- 172.21.15.32:0/3240532632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 >> 172.21.15.32:0/3240532632 conn(0x55eaeb5d2b30 msgr2=0x55eaeb5d1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda1effd640 0 client.16680 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 shutdown_connections 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 1 -- 172.21.15.32:0/3240532632 wait complete. 2024-04-23T18:25:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.736+0000 7fda48876740 20 client.16680 trim_cache size 0 max 16384 2024-04-23T18:25:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 Processor -- start 2024-04-23T18:25:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 -- start start 2024-04-23T18:25:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5ed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda401b0520 con 0x55eaeb5d6f40 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda401addf0 con 0x55eaeb5d6620 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.765+0000 7fda48876740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fda401abc60 con 0x55eaeb5e1800 2024-04-23T18:25:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:37262/0 (socket says 172.21.15.32:37262) 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 -- 172.21.15.32:0/2044935098 learned_addr learned my addr 172.21.15.32:0/2044935098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 -- 172.21.15.32:0/2044935098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 msgr2=0x55eaeb5ed2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5ed2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 -- 172.21.15.32:0/2044935098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 msgr2=0x55eaeb5d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.766+0000 7fda3ffff640 1 -- 172.21.15.32:0/2044935098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaeb5d75b0 con 0x55eaeb5e1800 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.767+0000 7fda3ffff640 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fda340cdfa0 tx=0x7fda340053d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c32c623e23a330 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.767+0000 7fda257fa640 1 -- 172.21.15.32:0/2044935098 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3401c040 con 0x55eaeb5e1800 2024-04-23T18:25:04.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.767+0000 7fda257fa640 1 -- 172.21.15.32:0/2044935098 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340870c0 con 0x55eaeb5e1800 2024-04-23T18:25:04.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.767+0000 7fda257fa640 1 -- 172.21.15.32:0/2044935098 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34098db0 con 0x55eaeb5e1800 2024-04-23T18:25:04.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.768+0000 7fda48876740 1 -- 172.21.15.32:0/2044935098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 msgr2=0x55eaeb5f9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.768+0000 7fda48876740 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fda340cdfa0 tx=0x7fda340053d0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.771+0000 7fda48876740 1 -- 172.21.15.32:0/2044935098 shutdown_connections 2024-04-23T18:25:04.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.771+0000 7fda48876740 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e1800 0x55eaeb5f9500 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.772+0000 7fda48876740 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.772+0000 7fda48876740 1 --2- 172.21.15.32:0/2044935098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6620 0x55eaeb5ed2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.772+0000 7fda48876740 1 -- 172.21.15.32:0/2044935098 >> 172.21.15.32:0/2044935098 conn(0x55eaeb5d2b30 msgr2=0x55eaeb5d1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.776+0000 7fda48876740 1 -- 172.21.15.32:0/2044935098 shutdown_connections 2024-04-23T18:25:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.778+0000 7fda48876740 1 -- 172.21.15.32:0/2044935098 wait complete. 2024-04-23T18:25:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.778+0000 7fda48876740 1 -- start start 2024-04-23T18:25:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.779+0000 7fda48876740 1 Processor -- start 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e7310 0x55eaeb5e7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda401ac2b0 con 0x55eaeb5e1800 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda401ad720 con 0x55eaeb5d6f40 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fda401ac980 con 0x55eaeb5e7310 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.782+0000 7fda48876740 10 client.0 _ll_register_callbacks cb 0x7fda300094b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e7310 0x55eaeb5e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38230/0 (socket says 172.21.15.32:38230) 2024-04-23T18:25:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 -- 172.21.15.32:0/2273304537 learned_addr learned my addr 172.21.15.32:0/2273304537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:04.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 -- 172.21.15.32:0/2273304537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e7310 msgr2=0x55eaeb5e7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e7310 0x55eaeb5e7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.778 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 -- 172.21.15.32:0/2273304537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 msgr2=0x55eaeb5e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 -- 172.21.15.32:0/2273304537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaeb656d80 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.783+0000 7fda3f7fe640 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda38009bf0 tx=0x7fda38009a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e75f4a72a7c0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda1effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3804e070 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8484+0+0 (secure 0 0 0) 0x7fda38003d80 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda340870c0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fda3805a050 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda48876740 10 client.26237 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55eaeb6580d0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.784+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340870c0 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fda38004310 con 0x55eaeb5d6f40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 10 client.26237 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 20 client.26237 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb658760 0x55eaeb65ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 4 client.26237 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eaeb543200 con 0x55eaeb658760 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda3effd640 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb658760 0x55eaeb65ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda1effd640 10 client.26237 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.785+0000 7fda3effd640 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb658760 0x55eaeb65ab80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9efe72eb2a5e2935 server_cookie=1c20cdf7eae1d031 in_seq=0 out_seq=0 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda1effd640 1 -- 172.21.15.32:0/2273304537 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eaeb543200 con 0x55eaeb658760 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda1effd640 10 client.26237 handle_command_reply: tid=0 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 client.26237 shutdown 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 msgr2=0x55eaeb5e9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda38009bf0 tx=0x7fda38009a10 comp rx=0 tx=0).stop 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 shutdown_connections 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55eaeb658760 0x55eaeb65ab80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eaeb5e7310 0x55eaeb5e7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eaeb5e1800 0x55eaeb5e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 --2- 172.21.15.32:0/2273304537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eaeb5d6f40 0x55eaeb5e9a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 >> 172.21.15.32:0/2273304537 conn(0x55eaeb5d2b30 msgr2=0x55eaeb5d1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.805+0000 7fda1effd640 0 client.26237 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.806+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 shutdown_connections 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.806+0000 7fda48876740 1 -- 172.21.15.32:0/2273304537 wait complete. 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.806+0000 7fda48876740 20 client.26237 trim_cache size 0 max 16384 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:25:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.807+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 msgr2=0x7fda280a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.807+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 0x7fda280a61d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda40101140 tx=0x7fda300156d0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.807+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 msgr2=0x7fda40198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.807+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 0x7fda40198b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda3400e970 tx=0x7fda3400ee30 comp rx=0 tx=0).stop 2024-04-23T18:25:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 shutdown_connections 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda40198660 0x7fda40198b10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda40107270 0x7fda4019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fda280a3d10 0x7fda280a61d0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda40101140 tx=0x7fda300156d0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 --2- 172.21.15.32:0/2734147098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda401021a0 0x7fda4019e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 >> 172.21.15.32:0/2734147098 conn(0x7fda400fda20 msgr2=0x7fda400fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 shutdown_connections 2024-04-23T18:25:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:04.808+0000 7fda45b5d640 1 -- 172.21.15.32:0/2734147098 wait complete. 2024-04-23T18:25:05.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:04 smithi040 ceph-mon[32626]: pgmap v308: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-23T18:25:05.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:04 smithi064 ceph-mon[33062]: pgmap v308: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-23T18:25:05.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:04 smithi032 ceph-mon[29221]: pgmap v308: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-23T18:25:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:06 smithi040 ceph-mon[32626]: pgmap v309: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-23T18:25:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:06 smithi064 ceph-mon[33062]: pgmap v309: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-23T18:25:07.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:06 smithi032 ceph-mon[29221]: pgmap v309: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-23T18:25:08.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:07 smithi064 ceph-mon[33062]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:25:08.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:07 smithi032 ceph-mon[29221]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:25:08.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:07 smithi040 ceph-mon[32626]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:25:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 580.0 s, 2205.6 tps, lat 46.818 ms stddev 64.438 2024-04-23T18:25:09.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:08 smithi064 ceph-mon[33062]: pgmap v311: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-04-23T18:25:09.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:08 smithi032 ceph-mon[29221]: pgmap v311: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-04-23T18:25:09.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:08 smithi040 ceph-mon[32626]: pgmap v311: 97 pgs: 97 active+clean; 4.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-04-23T18:25:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:10 smithi064 ceph-mon[33062]: pgmap v312: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.86k op/s 2024-04-23T18:25:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:10 smithi064 ceph-mon[33062]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:25:11.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:10 smithi032 ceph-mon[29221]: pgmap v312: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.86k op/s 2024-04-23T18:25:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:10 smithi032 ceph-mon[29221]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:25:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:10 smithi040 ceph-mon[32626]: pgmap v312: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.86k op/s 2024-04-23T18:25:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:10 smithi040 ceph-mon[32626]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:25:13.167 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:12 smithi040 ceph-mon[32626]: pgmap v314: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 2.01k op/s 2024-04-23T18:25:13.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:12 smithi064 ceph-mon[33062]: pgmap v314: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 2.01k op/s 2024-04-23T18:25:13.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:12 smithi032 ceph-mon[29221]: pgmap v314: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 2.01k op/s 2024-04-23T18:25:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 585.0 s, 2374.0 tps, lat 37.962 ms stddev 44.212 2024-04-23T18:25:15.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:14 smithi064 ceph-mon[33062]: pgmap v315: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.8 MiB/s wr, 452 op/s 2024-04-23T18:25:15.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:14 smithi032 ceph-mon[29221]: pgmap v315: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.8 MiB/s wr, 452 op/s 2024-04-23T18:25:15.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:14 smithi040 ceph-mon[32626]: pgmap v315: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.8 MiB/s wr, 452 op/s 2024-04-23T18:25:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:16 smithi064 ceph-mon[33062]: pgmap v316: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 1.75k op/s 2024-04-23T18:25:17.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:16 smithi032 ceph-mon[29221]: pgmap v316: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 1.75k op/s 2024-04-23T18:25:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:16 smithi040 ceph-mon[32626]: pgmap v316: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 1.75k op/s 2024-04-23T18:25:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.957+0000 7f3012d4f640 1 -- 172.21.15.32:0/4252822809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c107280 msgr2=0x7f300c10ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.957+0000 7f3012d4f640 1 --2- 172.21.15.32:0/4252822809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c107280 0x7f300c10ee50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc01a910 tx=0x7f2ffc03b450 comp rx=0 tx=0).stop 2024-04-23T18:25:17.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.959+0000 7f3012d4f640 1 -- 172.21.15.32:0/4252822809 shutdown_connections 2024-04-23T18:25:17.953 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.959+0000 7f3012d4f640 1 --2- 172.21.15.32:0/4252822809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c107280 0x7f300c10ee50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.959+0000 7f3012d4f640 1 --2- 172.21.15.32:0/4252822809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c102b60 0x7f300c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.959+0000 7f3012d4f640 1 --2- 172.21.15.32:0/4252822809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f300c1021b0 0x7f300c102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.959+0000 7f3012d4f640 1 -- 172.21.15.32:0/4252822809 >> 172.21.15.32:0/4252822809 conn(0x7f300c0fda30 msgr2=0x7f300c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.960+0000 7f3012d4f640 1 -- 172.21.15.32:0/4252822809 shutdown_connections 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.960+0000 7f3012d4f640 1 -- 172.21.15.32:0/4252822809 wait complete. 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.961+0000 7f3012d4f640 1 Processor -- start 2024-04-23T18:25:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.961+0000 7f3012d4f640 1 -- start start 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c1021b0 0x7f300c074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f300c107280 0x7f300c074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f300c06ccd0 con 0x7f300c1021b0 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f300c06cb50 con 0x7f300c107280 2024-04-23T18:25:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3012d4f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f300c06ce50 con 0x7f300c06e570 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f30112ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f30112ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53860/0 (socket says 172.21.15.32:53860) 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f30112ec640 1 -- 172.21.15.32:0/3362992226 learned_addr learned my addr 172.21.15.32:0/3362992226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.962+0000 7f3010aeb640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c1021b0 0x7f300c074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.963+0000 7f30112ec640 1 -- 172.21.15.32:0/3362992226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f300c107280 msgr2=0x7f300c074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.963+0000 7f30112ec640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f300c107280 0x7f300c074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.963+0000 7f30112ec640 1 -- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c1021b0 msgr2=0x7f300c074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.963+0000 7f30112ec640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c1021b0 0x7f300c074300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.963+0000 7f30112ec640 1 -- 172.21.15.32:0/3362992226 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300c1ab6f0 con 0x7f300c06e570 2024-04-23T18:25:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f30112ec640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc004b90 tx=0x7f2ffc004bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f3009ffb640 1 -- 172.21.15.32:0/3362992226 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc04f070 con 0x7f300c06e570 2024-04-23T18:25:17.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f3009ffb640 1 -- 172.21.15.32:0/3362992226 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ffc040bd0 con 0x7f300c06e570 2024-04-23T18:25:17.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f3009ffb640 1 -- 172.21.15.32:0/3362992226 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc046030 con 0x7f300c06e570 2024-04-23T18:25:17.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300c1ab8c0 con 0x7f300c06e570 2024-04-23T18:25:17.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.964+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f300c1abc70 con 0x7f300c06e570 2024-04-23T18:25:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.967+0000 7f3009ffb640 1 -- 172.21.15.32:0/3362992226 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2ffc052070 con 0x7f300c06e570 2024-04-23T18:25:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 Processor -- start 2024-04-23T18:25:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 -- start start 2024-04-23T18:25:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3009ffb640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 0x7f2ff40a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82620 0x561a8cd82a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3009ffb640 1 -- 172.21.15.32:0/3362992226 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f2ffc101d30 con 0x7f300c06e570 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd8d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a8cce7970 con 0x561a8cd82f40 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a8cc844a0 con 0x561a8cd82620 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f300bfff640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 0x7f2ff40a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.968+0000 7f3015a6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a8cce3930 con 0x561a8cd8d800 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f300bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd8d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53876/0 (socket says 172.21.15.32:53876) 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 -- 172.21.15.32:0/2905701495 learned_addr learned my addr 172.21.15.32:0/2905701495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f3010aeb640 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82620 0x561a8cd82a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 -- 172.21.15.32:0/2905701495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82620 msgr2=0x561a8cd82a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82620 0x561a8cd82a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 -- 172.21.15.32:0/2905701495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 msgr2=0x561a8cd8d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd8d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 -- 172.21.15.32:0/2905701495 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a8c9fa210 con 0x561a8cd8d800 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f300bfff640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 0x7f2ff40a6320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f300c06fb30 tx=0x7f30040138e0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f30112ec640 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0ca060 tx=0x7f2ffc0cd900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d87ac0c164222afa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f2ff2ffd640 1 -- 172.21.15.32:0/2905701495 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc059040 con 0x561a8cd8d800 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f2ff2ffd640 1 -- 172.21.15.32:0/2905701495 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ffc0ece50 con 0x561a8cd8d800 2024-04-23T18:25:17.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.969+0000 7f2ff2ffd640 1 -- 172.21.15.32:0/2905701495 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc042b10 con 0x561a8cd8d800 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 -- 172.21.15.32:0/2905701495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 msgr2=0x561a8cd8fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0ca060 tx=0x7f2ffc0cd900 comp rx=0 tx=0).stop 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 -- 172.21.15.32:0/2905701495 shutdown_connections 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd8fbf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd8d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2905701495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82620 0x561a8cd82a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.970+0000 7f3015a6f740 1 -- 172.21.15.32:0/2905701495 >> 172.21.15.32:0/2905701495 conn(0x561a8cd7eb30 msgr2=0x561a8cd7ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.971+0000 7f3015a6f740 1 -- 172.21.15.32:0/2905701495 shutdown_connections 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.971+0000 7f3015a6f740 1 -- 172.21.15.32:0/2905701495 wait complete. 2024-04-23T18:25:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.971+0000 7f3015a6f740 1 -- start start 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 Processor -- start 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 0x561a8cd8a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cd8aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a8cd901f0 con 0x561a8cd82620 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3010aeb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a8cc820b0 con 0x561a8cd8d800 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a8cd90370 con 0x561a8cd82f40 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3010aeb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33646/0 (socket says 172.21.15.32:33646) 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3010aeb640 1 -- 172.21.15.32:0/2143156249 learned_addr learned my addr 172.21.15.32:0/2143156249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f300bfff640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 0x561a8cd8a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f3015a6f740 10 client.0 _ll_register_callbacks cb 0x561a8cc6fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.972+0000 7f30112ec640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cd8aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 -- 172.21.15.32:0/2143156249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 msgr2=0x561a8cd8a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 0x561a8cd8a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 -- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 msgr2=0x561a8cd8aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cd8aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 -- 172.21.15.32:0/2143156249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a8cac6620 con 0x561a8cd82620 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f300bfff640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 0x561a8cd8a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3010aeb640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3000009fa0 tx=0x7f30000116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f89f72cb75ee29b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f2febfff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000019070 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ffc0ece50 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.973+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f3000015870 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3000025050 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 10 client.16692 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561a8cd9c120 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ffc0ece50 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f3000014b80 con 0x561a8cd82620 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 10 client.16692 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 20 client.16692 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8cd9d590 0x561a8cd9f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 4 client.16692 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.974+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a8ccf5390 con 0x561a8cd9d590 2024-04-23T18:25:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.976+0000 7f300bfff640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8cd9d590 0x561a8cd9f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.978+0000 7f300bfff640 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8cd9d590 0x561a8cd9f9a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3476bf30ee2a1047 server_cookie=a41604fc32ed1a58 in_seq=0 out_seq=0 2024-04-23T18:25:17.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.978+0000 7f2febfff640 10 client.16692 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f2febfff640 1 -- 172.21.15.32:0/2143156249 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a8ccf5390 con 0x561a8cd9d590 2024-04-23T18:25:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f2febfff640 10 client.16692 handle_command_reply: tid=0 2024-04-23T18:25:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 client.16692 shutdown 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 msgr2=0x561a8cd8d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3000009fa0 tx=0x7f30000116f0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 shutdown_connections 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8cd9d590 0x561a8cd9f9a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cd8aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82f40 0x561a8cd8a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 --2- 172.21.15.32:0/2143156249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd8d1b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 >> 172.21.15.32:0/2143156249 conn(0x561a8cd7eb30 msgr2=0x561a8cd848b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.979+0000 7f2febfff640 0 client.16692 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.980+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 shutdown_connections 2024-04-23T18:25:17.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.980+0000 7f3015a6f740 1 -- 172.21.15.32:0/2143156249 wait complete. 2024-04-23T18:25:17.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:17.980+0000 7f3015a6f740 20 client.16692 trim_cache size 0 max 16384 2024-04-23T18:25:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 Processor -- start 2024-04-23T18:25:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 -- start start 2024-04-23T18:25:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82620 0x561a8cd98f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cda52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b6a30 con 0x561a8cd82f40 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b4300 con 0x561a8cd8d800 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3015a6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b2170 con 0x561a8cd82620 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f300bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f300bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33650/0 (socket says 172.21.15.32:33650) 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f300bfff640 1 -- 172.21.15.32:0/3807763161 learned_addr learned my addr 172.21.15.32:0/3807763161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f3010aeb640 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82620 0x561a8cd98f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.001+0000 7f30112ec640 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cda52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 -- 172.21.15.32:0/3807763161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82620 msgr2=0x561a8cd98f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82620 0x561a8cd98f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 -- 172.21.15.32:0/3807763161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 msgr2=0x561a8cda52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cda52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 -- 172.21.15.32:0/3807763161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a8cd7a2e0 con 0x561a8cd82f40 2024-04-23T18:25:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f300bfff640 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f300c06fbc0 tx=0x7f3004019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3ebc1f3066dc2e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f2ff27fc640 1 -- 172.21.15.32:0/3807763161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300401daf0 con 0x561a8cd82f40 2024-04-23T18:25:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f2ff27fc640 1 -- 172.21.15.32:0/3807763161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300402f070 con 0x561a8cd82f40 2024-04-23T18:25:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.002+0000 7f2ff27fc640 1 -- 172.21.15.32:0/3807763161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004028960 con 0x561a8cd82f40 2024-04-23T18:25:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- 172.21.15.32:0/3807763161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 msgr2=0x561a8cd7e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f300c06fbc0 tx=0x7f3004019000 comp rx=0 tx=0).stop 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- 172.21.15.32:0/3807763161 shutdown_connections 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd8d800 0x561a8cda52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82f40 0x561a8cd7e430 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3807763161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd82620 0x561a8cd98f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- 172.21.15.32:0/3807763161 >> 172.21.15.32:0/3807763161 conn(0x561a8cd7eb30 msgr2=0x561a8cd98870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- 172.21.15.32:0/3807763161 shutdown_connections 2024-04-23T18:25:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- 172.21.15.32:0/3807763161 wait complete. 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 -- start start 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.003+0000 7f3015a6f740 1 Processor -- start 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd955f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd92fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f300bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f300bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:34760/0 (socket says 172.21.15.32:34760) 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f300bfff640 1 -- 172.21.15.32:0/3861443254 learned_addr learned my addr 172.21.15.32:0/3861443254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f30112ec640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd92fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b27c0 con 0x561a8cd82620 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b3c30 con 0x561a8cd82f40 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1b2e90 con 0x561a8cd8d800 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.004+0000 7f3015a6f740 10 client.0 _ll_register_callbacks cb 0x7f3000009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 -- 172.21.15.32:0/3861443254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 msgr2=0x561a8cd92fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd92fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 -- 172.21.15.32:0/3861443254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 msgr2=0x561a8cd955f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd955f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a8cd93510 con 0x561a8cd82f40 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f300bfff640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3004005fb0 tx=0x7f3004029430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994ce469a91ead6f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f2febfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300401d8f0 con 0x561a8cd82f40 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.005+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f300401da90 con 0x561a8cd82f40 2024-04-23T18:25:18.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300402f070 con 0x561a8cd82f40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f300401dc90 con 0x561a8cd82f40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 10 client.26249 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561a8ce03830 con 0x561a8cd82f40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300402f070 con 0x561a8cd82f40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f300400f040 con 0x561a8cd82f40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 10 client.26249 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.006+0000 7f3015a6f740 20 client.26249 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8ce03fa0 0x561a8ce063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f3015a6f740 4 client.26249 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a8ccef200 con 0x561a8ce03fa0 2024-04-23T18:25:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f3010aeb640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8ce03fa0 0x561a8ce063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f3010aeb640 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8ce03fa0 0x561a8ce063c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a44c6e891aaac452 server_cookie=1d2cf853f21323dc in_seq=0 out_seq=0 2024-04-23T18:25:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.007+0000 7f2febfff640 10 client.26249 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.020+0000 7f2febfff640 1 -- 172.21.15.32:0/3861443254 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561a8ccef200 con 0x561a8ce03fa0 2024-04-23T18:25:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f2febfff640 10 client.26249 handle_command_reply: tid=0 2024-04-23T18:25:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 client.26249 shutdown 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 msgr2=0x561a8cd92a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3004005fb0 tx=0x7f3004029430 comp rx=0 tx=0).stop 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 shutdown_connections 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a8ce03fa0 0x561a8ce063c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a8cd8d800 0x561a8cd92fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a8cd82f40 0x561a8cd92a90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 --2- 172.21.15.32:0/3861443254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a8cd82620 0x561a8cd955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f2febfff640 0 client.26249 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.021+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 >> 172.21.15.32:0/3861443254 conn(0x561a8cd7eb30 msgr2=0x561a8cd7d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 shutdown_connections 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3015a6f740 1 -- 172.21.15.32:0/3861443254 wait complete. 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3015a6f740 20 client.26249 trim_cache size 0 max 16384 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:25:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 msgr2=0x7f2ff40a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 0x7f2ff40a6320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f300c06fb30 tx=0x7f30040138e0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 msgr2=0x7f300c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.022+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc004b90 tx=0x7f2ffc004bc0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 shutdown_connections 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f300c06e570 0x7f300c06ea20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f300c107280 0x7f300c074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ff40a3e60 0x7f2ff40a6320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 --2- 172.21.15.32:0/3362992226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f300c1021b0 0x7f300c074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 >> 172.21.15.32:0/3362992226 conn(0x7f300c0fda30 msgr2=0x7f300c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 shutdown_connections 2024-04-23T18:25:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:18.023+0000 7f3012d4f640 1 -- 172.21.15.32:0/3362992226 wait complete. 2024-04-23T18:25:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 590.0 s, 2236.8 tps, lat 49.118 ms stddev 88.792 2024-04-23T18:25:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:18 smithi064 ceph-mon[33062]: pgmap v317: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 1.52k op/s 2024-04-23T18:25:19.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:18 smithi032 ceph-mon[29221]: pgmap v317: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 1.52k op/s 2024-04-23T18:25:19.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:18 smithi040 ceph-mon[32626]: pgmap v317: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 1.52k op/s 2024-04-23T18:25:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:20 smithi064 ceph-mon[33062]: pgmap v318: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 MiB/s wr, 1.81k op/s 2024-04-23T18:25:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:20 smithi032 ceph-mon[29221]: pgmap v318: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 MiB/s wr, 1.81k op/s 2024-04-23T18:25:21.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:20 smithi040 ceph-mon[32626]: pgmap v318: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 MiB/s wr, 1.81k op/s 2024-04-23T18:25:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:22 smithi064 ceph-mon[33062]: pgmap v319: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 MiB/s wr, 1.66k op/s 2024-04-23T18:25:23.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:22 smithi032 ceph-mon[29221]: pgmap v319: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 MiB/s wr, 1.66k op/s 2024-04-23T18:25:23.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:22 smithi040 ceph-mon[32626]: pgmap v319: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 MiB/s wr, 1.66k op/s 2024-04-23T18:25:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 595.0 s, 2277.0 tps, lat 43.431 ms stddev 46.066 2024-04-23T18:25:24.134 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 12.0 2024-04-23T18:25:24.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:25:24.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:23 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:24.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:23 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.301+0000 7fbabaffd640 1 -- 172.21.15.32:0/2822238591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0035020 con 0x7fbabc116f50 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 -- 172.21.15.32:0/2822238591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc116f50 msgr2=0x7fbabc119340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 --2- 172.21.15.32:0/2822238591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc116f50 0x7fbabc119340 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009900 tx=0x7fbab002a440 comp rx=0 tx=0).stop 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 -- 172.21.15.32:0/2822238591 shutdown_connections 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 --2- 172.21.15.32:0/2822238591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc116f50 0x7fbabc119340 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 --2- 172.21.15.32:0/2822238591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbabc109490 0x7fbabc1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 --2- 172.21.15.32:0/2822238591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc10c460 0x7fbabc108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 -- 172.21.15.32:0/2822238591 >> 172.21.15.32:0/2822238591 conn(0x7fbabc06ceb0 msgr2=0x7fbabc06d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 -- 172.21.15.32:0/2822238591 shutdown_connections 2024-04-23T18:25:24.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.302+0000 7fbac13bc640 1 -- 172.21.15.32:0/2822238591 wait complete. 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.303+0000 7fbac13bc640 1 Processor -- start 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.303+0000 7fbac13bc640 1 -- start start 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbabc109490 0x7fbabc10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc116f50 0x7fbabc10d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc11cb30 con 0x7fbabc10c460 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc11c9b0 con 0x7fbabc109490 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbac13bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc11ccb0 con 0x7fbabc116f50 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbabb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbabb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33666/0 (socket says 172.21.15.32:33666) 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.304+0000 7fbabb7fe640 1 -- 172.21.15.32:0/709055449 learned_addr learned my addr 172.21.15.32:0/709055449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbac0bbb640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc116f50 0x7fbabc10d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbabb7fe640 1 -- 172.21.15.32:0/709055449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc116f50 msgr2=0x7fbabc10d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbabb7fe640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc116f50 0x7fbabc10d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbabb7fe640 1 -- 172.21.15.32:0/709055449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbabc109490 msgr2=0x7fbabc10cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbabb7fe640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbabc109490 0x7fbabc10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.305+0000 7fbabb7fe640 1 -- 172.21.15.32:0/709055449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1b3780 con 0x7fbabc10c460 2024-04-23T18:25:24.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbabb7fe640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b480 tx=0x7fbaac00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac014070 con 0x7fbabc10c460 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc1b3a70 con 0x7fbabc10c460 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1b3fc0 con 0x7fbabc10c460 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaac010d20 con 0x7fbabc10c460 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.306+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac01e060 con 0x7fbabc10c460 2024-04-23T18:25:24.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.307+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa8005290 con 0x7fbabc10c460 2024-04-23T18:25:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.309+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fbaac017070 con 0x7fbabc10c460 2024-04-23T18:25:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.310+0000 7fbab97fa640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 0x7fbaa40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.310+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7fbaac0cfc40 con 0x7fbabc10c460 2024-04-23T18:25:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.311+0000 7fbabbfff640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 0x7fbaa40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.311+0000 7fbabbfff640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 0x7fbaa40a6250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbab400abb0 tx=0x7fbab4009250 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.316+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbaac09c440 con 0x7fbabc10c460 2024-04-23T18:25:24.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:23 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:24.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.692+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaa8005850 con 0x7fbabc10c460 2024-04-23T18:25:24.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.693+0000 7fbab97fa640 1 -- 172.21.15.32:0/709055449 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11851 (secure 0 0 0) 0x7fbaac09bd40 con 0x7fbabc10c460 2024-04-23T18:25:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:25:24.688 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:24:30.869819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:25:24.688 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-04-23T18:25:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.697+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 msgr2=0x7fbaa40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.697+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 0x7fbaa40a6250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbab400abb0 tx=0x7fbab4009250 comp rx=0 tx=0).stop 2024-04-23T18:25:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.697+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 msgr2=0x7fbabc10d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.697+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b480 tx=0x7fbaac00b940 comp rx=0 tx=0).stop 2024-04-23T18:25:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 shutdown_connections 2024-04-23T18:25:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbabc116f50 0x7fbabc10d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbaa40a3d90 0x7fbaa40a6250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc10c460 0x7fbabc10d110 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 --2- 172.21.15.32:0/709055449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbabc109490 0x7fbabc10cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.702+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 >> 172.21.15.32:0/709055449 conn(0x7fbabc06ceb0 msgr2=0x7fbabc072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.709+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 shutdown_connections 2024-04-23T18:25:24.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.711+0000 7fbac13bc640 1 -- 172.21.15.32:0/709055449 wait complete. 2024-04-23T18:25:24.739 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 72.0 --await-for 42.0' 2024-04-23T18:25:24.740 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 72.0 --await-for 42.0 2024-04-23T18:25:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 -- 172.21.15.32:0/2055918323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 msgr2=0x7f15c010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2055918323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c010dc70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009ca0 tx=0x7f15b802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 -- 172.21.15.32:0/2055918323 shutdown_connections 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2055918323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c010dc70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2055918323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c01010a0 0x7f15c0101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2055918323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c0106ed0 0x7f15c01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 -- 172.21.15.32:0/2055918323 >> 172.21.15.32:0/2055918323 conn(0x7f15c0077c40 msgr2=0x7f15c0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 -- 172.21.15.32:0/2055918323 shutdown_connections 2024-04-23T18:25:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.950+0000 7f15c64a3640 1 -- 172.21.15.32:0/2055918323 wait complete. 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 Processor -- start 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 -- start start 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 0x7f15c019f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 0x7f15c010cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15c0112200 con 0x7f15c0101a40 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15c0112080 con 0x7f15c0106ed0 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c64a3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15c0112380 con 0x7f15c01010a0 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33692/0 (socket says 172.21.15.32:33692) 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15bffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 0x7f15c019f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15bf7fe640 1 -- 172.21.15.32:0/2793949618 learned_addr learned my addr 172.21.15.32:0/2793949618 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.951+0000 7f15c4a40640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 0x7f15c010cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 -- 172.21.15.32:0/2793949618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 msgr2=0x7f15c019f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 0x7f15c019f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 -- 172.21.15.32:0/2793949618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 msgr2=0x7f15c010cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 0x7f15c010cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 -- 172.21.15.32:0/2793949618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c010d7c0 con 0x7f15c0101a40 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bffff640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 0x7f15c019f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15c4a40640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 0x7f15c010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9f0 tx=0x7f15b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bd7fa640 1 -- 172.21.15.32:0/2793949618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b400cd80 con 0x7f15c0101a40 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c010dab0 con 0x7f15c0101a40 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bd7fa640 1 -- 172.21.15.32:0/2793949618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b40108e0 con 0x7f15c0101a40 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c019af60 con 0x7f15c0101a40 2024-04-23T18:25:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.952+0000 7f15bd7fa640 1 -- 172.21.15.32:0/2793949618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4018980 con 0x7f15c0101a40 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.955+0000 7f15bd7fa640 1 -- 172.21.15.32:0/2793949618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f15b4010430 con 0x7f15c0101a40 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.955+0000 7f15c91b7740 1 Processor -- start 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.955+0000 7f15c91b7740 1 -- start start 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bd7fa640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 0x7f15a80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc2480a0 0x55e3cc248480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc253090 0x55e3cc255480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc1ad370 con 0x55e3cc2489c0 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc14a2e0 con 0x55e3cc2480a0 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c91b7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc1a9330 con 0x55e3cc253090 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c4a40640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 0x7f15a80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33706/0 (socket says 172.21.15.32:33706) 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3785122022 learned_addr learned my addr 172.21.15.32:0/3785122022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bd7fa640 1 -- 172.21.15.32:0/2793949618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f15b40d0b70 con 0x7f15c0101a40 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15c4a40640 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc253090 0x55e3cc255480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3785122022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc253090 msgr2=0x55e3cc255480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc253090 0x55e3cc255480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3785122022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc2480a0 msgr2=0x55e3cc248480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc2480a0 0x55e3cc248480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.956+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3785122022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3cbf8bd80 con 0x55e3cc2489c0 2024-04-23T18:25:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.957+0000 7f15c4a40640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 0x7f15a80a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f15b8006010 tx=0x7f15b8005c80 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.957+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f15b40a5270 tx=0x7f15b4014cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deee1da34a02b0ec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.958+0000 7f15a67fc640 1 -- 172.21.15.32:0/3785122022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b409cb30 con 0x55e3cc2489c0 2024-04-23T18:25:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.958+0000 7f15a67fc640 1 -- 172.21.15.32:0/3785122022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b40a0ee0 con 0x55e3cc2489c0 2024-04-23T18:25:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.958+0000 7f15a67fc640 1 -- 172.21.15.32:0/3785122022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b40991a0 con 0x55e3cc2489c0 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- 172.21.15.32:0/3785122022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 msgr2=0x55e3cc252b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f15b40a5270 tx=0x7f15b4014cb0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- 172.21.15.32:0/3785122022 shutdown_connections 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc253090 0x55e3cc255480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc252b50 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3785122022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc2480a0 0x55e3cc248480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- 172.21.15.32:0/3785122022 >> 172.21.15.32:0/3785122022 conn(0x55e3cc2443c0 msgr2=0x55e3cc2447a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- 172.21.15.32:0/3785122022 shutdown_connections 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- 172.21.15.32:0/3785122022 wait complete. 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 -- start start 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 Processor -- start 2024-04-23T18:25:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.959+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc250070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc250af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc255e80 con 0x55e3cc2480a0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc147fb0 con 0x55e3cc253090 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e3cc256000 con 0x55e3cc2489c0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 10 client.0 _ll_register_callbacks cb 0x55e3cc1479a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc250070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53958/0 (socket says 172.21.15.32:53958) 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3584509066 learned_addr learned my addr 172.21.15.32:0/3584509066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bffff640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc250070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33710/0 (socket says 172.21.15.32:33710) 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c4a40640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc250af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 msgr2=0x55e3cc250af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc250af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3584509066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 msgr2=0x55e3cc250070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc250070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 -- 172.21.15.32:0/3584509066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3cbfec510 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9c0 tx=0x7f15b409d2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=400baf527e18ebfe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f159f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b409de60 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15b40a0ee0 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.960+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f15b4014d20 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f15b4025170 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 10 client.26278 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e3cbe7f670 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b40a0ee0 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f15b4018b20 con 0x55e3cc2489c0 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 10 client.26278 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 10 client.26278 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:25:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 20 client.26278 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2632f0 0x55e3cc2656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 4 client.26278 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3cc1bad90 con 0x55e3cc2632f0 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.961+0000 7f15bffff640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2632f0 0x55e3cc2656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.962+0000 7f159f7fe640 10 client.26278 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.962+0000 7f15bffff640 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2632f0 0x55e3cc2656e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bc95ca7d29f8130 server_cookie=3c6a5d8f64a03b40 in_seq=0 out_seq=0 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f159f7fe640 1 -- 172.21.15.32:0/3584509066 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3cc1bad90 con 0x55e3cc2632f0 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f159f7fe640 10 client.26278 handle_command_reply: tid=0 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 client.26278 shutdown 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 msgr2=0x55e3cc2505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9c0 tx=0x7f15b409d2f0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 shutdown_connections 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2632f0 0x55e3cc2656e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc250af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc2505b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f159f7fe640 0 client.26278 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3584509066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc250070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 >> 172.21.15.32:0/3584509066 conn(0x55e3cc2443c0 msgr2=0x55e3cc24a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 shutdown_connections 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 1 -- 172.21.15.32:0/3584509066 wait complete. 2024-04-23T18:25:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.963+0000 7f15c91b7740 20 client.26278 trim_cache size 0 max 16384 2024-04-23T18:25:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.989+0000 7f15c91b7740 1 Processor -- start 2024-04-23T18:25:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.989+0000 7f15c91b7740 1 -- start start 2024-04-23T18:25:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.989+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc268700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc268c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c91b7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01bf250 con 0x55e3cc2480a0 2024-04-23T18:25:24.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c91b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01bcb20 con 0x55e3cc253090 2024-04-23T18:25:24.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c91b7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01ba990 con 0x55e3cc2489c0 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33718/0 (socket says 172.21.15.32:33718) 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15bffff640 1 -- 172.21.15.32:0/664768706 learned_addr learned my addr 172.21.15.32:0/664768706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc268c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc268700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 -- 172.21.15.32:0/664768706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 msgr2=0x55e3cc268700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc268700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 -- 172.21.15.32:0/664768706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 msgr2=0x55e3cc24a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15c4a40640 1 -- 172.21.15.32:0/664768706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3cc24e5c0 con 0x55e3cc253090 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.990+0000 7f15bffff640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.991+0000 7f15c4a40640 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc268c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15b802ade0 tx=0x7f15b8048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17faab7eb437e9f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.991+0000 7f15a5ffb640 1 -- 172.21.15.32:0/664768706 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8004580 con 0x55e3cc253090 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.991+0000 7f15a5ffb640 1 -- 172.21.15.32:0/664768706 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b8036040 con 0x55e3cc253090 2024-04-23T18:25:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.991+0000 7f15a5ffb640 1 -- 172.21.15.32:0/664768706 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b803f6a0 con 0x55e3cc253090 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 -- 172.21.15.32:0/664768706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 msgr2=0x55e3cc268c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc268c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15b802ade0 tx=0x7f15b8048000 comp rx=0 tx=0).stop 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 -- 172.21.15.32:0/664768706 shutdown_connections 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc268c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2489c0 0x55e3cc268700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 --2- 172.21.15.32:0/664768706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2480a0 0x55e3cc24a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 -- 172.21.15.32:0/664768706 >> 172.21.15.32:0/664768706 conn(0x55e3cc2443c0 msgr2=0x55e3cc252130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 -- 172.21.15.32:0/664768706 shutdown_connections 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.992+0000 7f15c91b7740 1 -- 172.21.15.32:0/664768706 wait complete. 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.993+0000 7f15c91b7740 1 -- start start 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.993+0000 7f15c91b7740 1 Processor -- start 2024-04-23T18:25:24.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.993+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.993+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.993+0000 7f15c91b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc266ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c91b7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01bafe0 con 0x55e3cc2489c0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c91b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01bc450 con 0x55e3cc253090 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c91b7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01bb6b0 con 0x55e3cc2480a0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:53978/0 (socket says 172.21.15.32:53978) 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33720/0 (socket says 172.21.15.32:33720) 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 -- 172.21.15.32:0/3803396856 learned_addr learned my addr 172.21.15.32:0/3803396856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c4a40640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc266ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c91b7740 10 client.0 _ll_register_callbacks cb 0x7f15b40056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 -- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 msgr2=0x55e3cc266ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc266ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 -- 172.21.15.32:0/3803396856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 msgr2=0x55e3cc266960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 -- 172.21.15.32:0/3803396856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3cc2c8720 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15c4a40640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc266ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f15bffff640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55e3cc256080 tx=0x7f15b00139d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3506ffb168de7780 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.994+0000 7f159f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0004460 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f15b001f390 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15b8036040 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f15b001f590 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 10 client.26284 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e3cc2c9a20 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b8036040 con 0x55e3cc2480a0 2024-04-23T18:25:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f15b0010430 con 0x55e3cc2480a0 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 10 client.26284 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 10 client.26284 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 20 client.26284 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2ca190 0x55e3cc2cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 4 client.26284 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 72.0, "await_for": 42.0}] 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.995+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 72.0, "await_for": 42.0}) -- 0x55e3cc1b4c00 con 0x55e3cc2ca190 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.996+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2ca190 0x55e3cc2cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.997+0000 7f15bf7fe640 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2ca190 0x55e3cc2cc5b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=966c7df9ffd5be1b server_cookie=2cf85d4f178b1743 in_seq=0 out_seq=0 2024-04-23T18:25:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:24.997+0000 7f159f7fe640 10 client.26284 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:25.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:24 smithi032 ceph-mon[29221]: pgmap v320: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-23T18:25:25.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:24 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/709055449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:25.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:24 smithi064 ceph-mon[33062]: pgmap v320: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-23T18:25:25.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:24 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/709055449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f159f7fe640 1 -- 172.21.15.32:0/3803396856 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e3cc1b4c00 con 0x55e3cc2ca190 2024-04-23T18:25:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f159f7fe640 10 client.26284 handle_command_reply: tid=0 2024-04-23T18:25:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 client.26284 shutdown 2024-04-23T18:25:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 msgr2=0x55e3cc266420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55e3cc256080 tx=0x7f15b00139d0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 shutdown_connections 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e3cc2ca190 0x55e3cc2cc5b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3cc253090 0x55e3cc266ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f159f7fe640 0 client.26284 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3cc2489c0 0x55e3cc266960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 --2- 172.21.15.32:0/3803396856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3cc2480a0 0x55e3cc266420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.275+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 >> 172.21.15.32:0/3803396856 conn(0x55e3cc2443c0 msgr2=0x55e3cc242d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 shutdown_connections 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c91b7740 1 -- 172.21.15.32:0/3803396856 wait complete. 2024-04-23T18:25:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c91b7740 20 client.26284 trim_cache size 0 max 16384 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 40, 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 34, 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "6f37624f": { 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 34, 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:25:25.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 72.0, 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:25.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:25.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:25.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:25.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:25.274 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:25:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 msgr2=0x7f15a80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 0x7f15a80a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f15b8006010 tx=0x7f15b8005c80 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 msgr2=0x7f15c019f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.276+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9f0 tx=0x7f15b400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 shutdown_connections 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c0106ed0 0x7f15c010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f15a80a3d20 0x7f15a80a61e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15c0101a40 0x7f15c019f820 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 --2- 172.21.15.32:0/2793949618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c01010a0 0x7f15c019f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 >> 172.21.15.32:0/2793949618 conn(0x7f15c0077c40 msgr2=0x7f15c00ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 shutdown_connections 2024-04-23T18:25:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:25.278+0000 7f15c64a3640 1 -- 172.21.15.32:0/2793949618 wait complete. 2024-04-23T18:25:25.284 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6f37624f', quiesce took 1.1 seconds. Will release after: 12.0 2024-04-23T18:25:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:24 smithi040 ceph-mon[32626]: pgmap v320: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-23T18:25:25.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:24 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/709055449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: pgmap v321: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 MiB/s wr, 1.76k op/s 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:27.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: pgmap v321: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 MiB/s wr, 1.76k op/s 2024-04-23T18:25:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:27.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: pgmap v321: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 MiB/s wr, 1.76k op/s 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:27.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:25:28.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:25:28.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:25:28.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:25:28.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:25:28.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:25:28.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:27 smithi032 ceph-mon[29221]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:25:28.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:27 smithi040 ceph-mon[32626]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi032 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi064 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-23T18:25:28.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:27 smithi064 ceph-mon[33062]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:25:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 600.0 s, 375.7 tps, lat 64.584 ms stddev 73.830 2024-04-23T18:25:29.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:28 smithi032 ceph-mon[29221]: pgmap v322: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 792 op/s 2024-04-23T18:25:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:28 smithi040 ceph-mon[32626]: pgmap v322: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 792 op/s 2024-04-23T18:25:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:28 smithi064 ceph-mon[33062]: pgmap v322: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 792 op/s 2024-04-23T18:25:31.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:30 smithi032 ceph-mon[29221]: pgmap v323: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 22 MiB/s wr, 1.79k op/s 2024-04-23T18:25:31.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:30 smithi040 ceph-mon[32626]: pgmap v323: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 22 MiB/s wr, 1.79k op/s 2024-04-23T18:25:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- 172.21.15.32:0/2740360895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd410ec90 msgr2=0x7effd4111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 --2- 172.21.15.32:0/2740360895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd410ec90 0x7effd4111080 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7effcc009ca0 tx=0x7effcc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- 172.21.15.32:0/2740360895 shutdown_connections 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 --2- 172.21.15.32:0/2740360895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd410ec90 0x7effd4111080 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 --2- 172.21.15.32:0/2740360895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406aeb0 0x7effd410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 --2- 172.21.15.32:0/2740360895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd406a590 0x7effd406a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- 172.21.15.32:0/2740360895 >> 172.21.15.32:0/2740360895 conn(0x7effd4077d10 msgr2=0x7effd4078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- 172.21.15.32:0/2740360895 shutdown_connections 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- 172.21.15.32:0/2740360895 wait complete. 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 Processor -- start 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.127+0000 7effdb84e640 1 -- start start 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406a590 0x7effd41a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd410ec90 0x7effd419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7effd4114450 con 0x7effd406aeb0 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd41142d0 con 0x7effd406a590 2024-04-23T18:25:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effdb84e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd41145d0 con 0x7effd410ec90 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43608/0 (socket says 172.21.15.32:43608) 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 -- 172.21.15.32:0/3619553789 learned_addr learned my addr 172.21.15.32:0/3619553789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd95ea640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406a590 0x7effd41a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd9deb640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd410ec90 0x7effd419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 -- 172.21.15.32:0/3619553789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd410ec90 msgr2=0x7effd419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd410ec90 0x7effd419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 -- 172.21.15.32:0/3619553789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406a590 msgr2=0x7effd41a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406a590 0x7effd41a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 -- 172.21.15.32:0/3619553789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd419b050 con 0x7effd406aeb0 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.128+0000 7effd8de9640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7effc400e9f0 tx=0x7effc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.129+0000 7effca7fc640 1 -- 172.21.15.32:0/3619553789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc400cd80 con 0x7effd406aeb0 2024-04-23T18:25:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.129+0000 7effca7fc640 1 -- 172.21.15.32:0/3619553789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc40108e0 con 0x7effd406aeb0 2024-04-23T18:25:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.129+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd419b340 con 0x7effd406aeb0 2024-04-23T18:25:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.129+0000 7effca7fc640 1 -- 172.21.15.32:0/3619553789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4018980 con 0x7effd406aeb0 2024-04-23T18:25:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.129+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd41a7600 con 0x7effd406aeb0 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effca7fc640 1 -- 172.21.15.32:0/3619553789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7effc4010430 con 0x7effd406aeb0 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 Processor -- start 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 -- start start 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555b5c72f970 con 0x555b5c7caf40 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555b5c6cc4a0 con 0x555b5c7d5800 2024-04-23T18:25:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effde567740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b5c72b930 con 0x555b5c7ca620 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.132+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43612/0 (socket says 172.21.15.32:43612) 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd95ea640 1 -- 172.21.15.32:0/2430765081 learned_addr learned my addr 172.21.15.32:0/2430765081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effca7fc640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 0x7effc00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd8de9640 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 0x7effc00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 -- 172.21.15.32:0/2430765081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 msgr2=0x555b5c7d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 -- 172.21.15.32:0/2430765081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 msgr2=0x555b5c7d52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 -- 172.21.15.32:0/2430765081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b5c442210 con 0x555b5c7ca620 2024-04-23T18:25:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd95ea640 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effca7fc640 1 -- 172.21.15.32:0/3619553789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7effc40d0aa0 con 0x7effd406aeb0 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7caa00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7effcc005e30 tx=0x7effcc031e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4ece1a183f1a3f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.133+0000 7effd9deb640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 0x7effc00a61d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7effcc048000 tx=0x7effcc04a000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effbf7fe640 1 -- 172.21.15.32:0/2430765081 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc0098e0 con 0x555b5c7ca620 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effbf7fe640 1 -- 172.21.15.32:0/2430765081 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effcc04e020 con 0x555b5c7ca620 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effbf7fe640 1 -- 172.21.15.32:0/2430765081 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc004830 con 0x555b5c7ca620 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 -- 172.21.15.32:0/2430765081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 msgr2=0x555b5c7caa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7caa00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7effcc005e30 tx=0x7effcc031e40 comp rx=0 tx=0).stop 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 -- 172.21.15.32:0/2430765081 shutdown_connections 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 --2- 172.21.15.32:0/2430765081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7caa00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 -- 172.21.15.32:0/2430765081 >> 172.21.15.32:0/2430765081 conn(0x555b5c7c6b30 msgr2=0x555b5c7c6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 -- 172.21.15.32:0/2430765081 shutdown_connections 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.134+0000 7effde567740 1 -- 172.21.15.32:0/2430765081 wait complete. 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 -- start start 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 Processor -- start 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555b5c7d8250 con 0x555b5c7caf40 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555b5c6ca0b0 con 0x555b5c7d5800 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.135+0000 7effde567740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b5c7d83d0 con 0x555b5c7ca620 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effde567740 10 client.0 _ll_register_callbacks cb 0x555b5c6b7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd8de9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:47878/0 (socket says 172.21.15.32:47878) 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 -- 172.21.15.32:0/1167364428 learned_addr learned my addr 172.21.15.32:0/1167364428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd9deb640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 -- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 msgr2=0x555b5c7d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 -- 172.21.15.32:0/1167364428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 msgr2=0x555b5c7d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 -- 172.21.15.32:0/1167364428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b5c50e620 con 0x555b5c7ca620 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd8de9640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effd95ea640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7effd000bf40 tx=0x7effd00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482f680e9ff4d055 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:25:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0005020 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7effd00051c0 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.136+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effcc04e020 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7effd00053c0 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 10 client.26296 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555b5c7e4200 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effcc04e020 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7effd0021020 con 0x555b5c7ca620 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 10 client.26296 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 10 client.26296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 20 client.26296 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c7e5710 0x555b5c7e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 4 client.26296 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b5c73d390 con 0x555b5c7e5710 2024-04-23T18:25:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.137+0000 7effd8de9640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c7e5710 0x555b5c7e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.138+0000 7effd8de9640 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c7e5710 0x555b5c7e7b00 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a788e83d7f543b server_cookie=9363a697190fcf60 in_seq=0 out_seq=0 2024-04-23T18:25:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.138+0000 7effbcff9640 10 client.26296 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effbcff9640 1 -- 172.21.15.32:0/1167364428 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b5c73d390 con 0x555b5c7e5710 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effbcff9640 10 client.26296 handle_command_reply: tid=0 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 client.26296 shutdown 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 msgr2=0x555b5c7d5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7effd000bf40 tx=0x7effd00118d0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 shutdown_connections 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c7e5710 0x555b5c7e7b00 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7caf40 0x555b5c7d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 --2- 172.21.15.32:0/1167364428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7ca620 0x555b5c7d5200 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effbcff9640 0 client.26296 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 >> 172.21.15.32:0/1167364428 conn(0x555b5c7c6b30 msgr2=0x555b5c7cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 shutdown_connections 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 1 -- 172.21.15.32:0/1167364428 wait complete. 2024-04-23T18:25:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.139+0000 7effde567740 20 client.26296 trim_cache size 0 max 16384 2024-04-23T18:25:31.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 Processor -- start 2024-04-23T18:25:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 -- start start 2024-04-23T18:25:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7caf40 0x555b5c7d4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7d5800 0x555b5c7ed410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7effd41b25d0 con 0x555b5c7ca620 2024-04-23T18:25:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd41afea0 con 0x555b5c7caf40 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effde567740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd41add10 con 0x555b5c7d5800 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43632/0 (socket says 172.21.15.32:43632) 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.154+0000 7effd95ea640 1 -- 172.21.15.32:0/4257075699 learned_addr learned my addr 172.21.15.32:0/4257075699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd9deb640 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7d5800 0x555b5c7ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd8de9640 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7caf40 0x555b5c7d4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 -- 172.21.15.32:0/4257075699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7d5800 msgr2=0x555b5c7ed410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7d5800 0x555b5c7ed410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 -- 172.21.15.32:0/4257075699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7caf40 msgr2=0x555b5c7d4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7caf40 0x555b5c7d4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 -- 172.21.15.32:0/4257075699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b5c7d2e70 con 0x555b5c7ca620 2024-04-23T18:25:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effd95ea640 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7effd000c010 tx=0x7effd00058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c75593846961fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effbeffd640 1 -- 172.21.15.32:0/4257075699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0010ce0 con 0x555b5c7ca620 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effbeffd640 1 -- 172.21.15.32:0/4257075699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd0010e80 con 0x555b5c7ca620 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.155+0000 7effbeffd640 1 -- 172.21.15.32:0/4257075699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0023490 con 0x555b5c7ca620 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- 172.21.15.32:0/4257075699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 msgr2=0x555b5c7c6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7effd000c010 tx=0x7effd00058f0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- 172.21.15.32:0/4257075699 shutdown_connections 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7d5800 0x555b5c7ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7caf40 0x555b5c7d4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 --2- 172.21.15.32:0/4257075699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7c6430 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- 172.21.15.32:0/4257075699 >> 172.21.15.32:0/4257075699 conn(0x555b5c7c6b30 msgr2=0x555b5c7cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- 172.21.15.32:0/4257075699 shutdown_connections 2024-04-23T18:25:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- 172.21.15.32:0/4257075699 wait complete. 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 -- start start 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.156+0000 7effde567740 1 Processor -- start 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7caf40 0x555b5c7cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7effd41ae360 con 0x555b5c7ca620 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd41af7d0 con 0x555b5c7d5800 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd41aea30 con 0x555b5c7caf40 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effde567740 10 client.0 _ll_register_callbacks cb 0x7effd000ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43648/0 (socket says 172.21.15.32:43648) 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 -- 172.21.15.32:0/467082826 learned_addr learned my addr 172.21.15.32:0/467082826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd8de9640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7caf40 0x555b5c7cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd9deb640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 -- 172.21.15.32:0/467082826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7caf40 msgr2=0x555b5c7cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7caf40 0x555b5c7cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 -- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 msgr2=0x555b5c7cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 -- 172.21.15.32:0/467082826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b5c84ac80 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effd95ea640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7effd0010930 tx=0x7effd000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f354db4a42c24ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0005210 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.157+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7effd00035f0 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 1 -- 172.21.15.32:0/467082826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b5c737200 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7effd0043050 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 10 client.16737 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 1 -- 172.21.15.32:0/467082826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555b5c84bdf0 con 0x555b5c7ca620 2024-04-23T18:25:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555b5c737200 con 0x555b5c7ca620 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7effd0023ad0 con 0x555b5c7ca620 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 10 client.16737 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 20 client.16737 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c84c4b0 0x555b5c84e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 4 client.16737 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effde567740 1 -- 172.21.15.32:0/467082826 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b5c6b35f0 con 0x555b5c84c4b0 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.158+0000 7effd8de9640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c84c4b0 0x555b5c84e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.159+0000 7effd8de9640 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c84c4b0 0x555b5c84e8d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0cdbe0923baf045 server_cookie=5b37c368671e59df in_seq=0 out_seq=0 2024-04-23T18:25:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.159+0000 7effbcff9640 10 client.16737 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.174+0000 7effbcff9640 1 -- 172.21.15.32:0/467082826 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b5c6b35f0 con 0x555b5c84c4b0 2024-04-23T18:25:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.174+0000 7effbcff9640 10 client.16737 handle_command_reply: tid=0 2024-04-23T18:25:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.174+0000 7effde567740 1 client.16737 shutdown 2024-04-23T18:25:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 -- 172.21.15.32:0/467082826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 msgr2=0x555b5c7ddb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7effd0010930 tx=0x7effd000bac0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 -- 172.21.15.32:0/467082826 shutdown_connections 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555b5c84c4b0 0x555b5c84e8d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555b5c7d5800 0x555b5c7cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b5c7caf40 0x555b5c7cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 --2- 172.21.15.32:0/467082826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b5c7ca620 0x555b5c7ddb60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effbcff9640 0 client.16737 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 -- 172.21.15.32:0/467082826 >> 172.21.15.32:0/467082826 conn(0x555b5c7c6b30 msgr2=0x555b5c7cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 -- 172.21.15.32:0/467082826 shutdown_connections 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 1 -- 172.21.15.32:0/467082826 wait complete. 2024-04-23T18:25:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effde567740 20 client.16737 trim_cache size 0 max 16384 2024-04-23T18:25:31.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:25:31.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:25:31.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:25:31.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 msgr2=0x7effc00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 0x7effc00a61d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7effcc048000 tx=0x7effcc04a000 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 msgr2=0x7effd41a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7effc400e9f0 tx=0x7effc400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.175+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 shutdown_connections 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd410ec90 0x7effd419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7effc00a3d10 0x7effc00a61d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effd406aeb0 0x7effd41a0bc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 --2- 172.21.15.32:0/3619553789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd406a590 0x7effd41a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 >> 172.21.15.32:0/3619553789 conn(0x7effd4077d10 msgr2=0x7effd41101e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 shutdown_connections 2024-04-23T18:25:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:31.176+0000 7effdb84e640 1 -- 172.21.15.32:0/3619553789 wait complete. 2024-04-23T18:25:31.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:30 smithi064 ceph-mon[33062]: pgmap v323: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 22 MiB/s wr, 1.79k op/s 2024-04-23T18:25:33.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:32 smithi032 ceph-mon[29221]: pgmap v324: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:32 smithi040 ceph-mon[32626]: pgmap v324: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:33.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:32 smithi064 ceph-mon[33062]: pgmap v324: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:33.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 605.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:25:35.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:34 smithi032 ceph-mon[29221]: pgmap v325: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:34 smithi040 ceph-mon[32626]: pgmap v325: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:35.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:34 smithi064 ceph-mon[33062]: pgmap v325: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:36.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:35 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:36.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:35 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:36.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:35 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:36 smithi032 ceph-mon[29221]: pgmap v326: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:37.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:36 smithi040 ceph-mon[32626]: pgmap v326: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:36 smithi064 ceph-mon[33062]: pgmap v326: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-23T18:25:37.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:25:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- 172.21.15.32:0/4103856975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a44101a40 msgr2=0x7f2a4410dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/4103856975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a44101a40 0x7f2a4410dc70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c009ca0 tx=0x7f2a3c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- 172.21.15.32:0/4103856975 shutdown_connections 2024-04-23T18:25:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/4103856975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a44101a40 0x7f2a4410dc70 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/4103856975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441010a0 0x7f2a44101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/4103856975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a441072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- 172.21.15.32:0/4103856975 >> 172.21.15.32:0/4103856975 conn(0x7f2a44077c40 msgr2=0x7f2a44078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- 172.21.15.32:0/4103856975 shutdown_connections 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- 172.21.15.32:0/4103856975 wait complete. 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 Processor -- start 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.409+0000 7f2a4acfd640 1 -- start start 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44101a40 0x7f2a44198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a4419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44111090 con 0x7f2a441010a0 2024-04-23T18:25:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44110f10 con 0x7f2a44101a40 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4acfd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44111210 con 0x7f2a44106ed0 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45372/0 (socket says 172.21.15.32:45372) 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 -- 172.21.15.32:0/1128611917 learned_addr learned my addr 172.21.15.32:0/1128611917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a43fff640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44101a40 0x7f2a44198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4929a640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a4419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 -- 172.21.15.32:0/1128611917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 msgr2=0x7f2a4419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a4419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 -- 172.21.15.32:0/1128611917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44101a40 msgr2=0x7f2a44198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44101a40 0x7f2a44198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 -- 172.21.15.32:0/1128611917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4419d6d0 con 0x7f2a441010a0 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a4929a640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a4419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a48a99640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400d950 tx=0x7f2a3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.410+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34017030 con 0x7f2a441010a0 2024-04-23T18:25:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.411+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4419d9c0 con 0x7f2a441010a0 2024-04-23T18:25:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.411+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a34004020 con 0x7f2a441010a0 2024-04-23T18:25:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.411+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a340047a0 con 0x7f2a441010a0 2024-04-23T18:25:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.411+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a441a5250 con 0x7f2a441010a0 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.412+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a44068e00 con 0x7f2a441010a0 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.413+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2a34009a30 con 0x7f2a441010a0 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.413+0000 7f2a41ffb640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 0x7f2a2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.413+0000 7f2a43fff640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 0x7f2a2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.413+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f2a340d00f0 con 0x7f2a441010a0 2024-04-23T18:25:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.414+0000 7f2a43fff640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 0x7f2a2c0a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2a440781d0 tx=0x7f2a38005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.418+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a3409c850 con 0x7f2a441010a0 2024-04-23T18:25:37.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.647+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a44199a30 con 0x7f2a441010a0 2024-04-23T18:25:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.648+0000 7f2a41ffb640 1 -- 172.21.15.32:0/1128611917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11852 (secure 0 0 0) 0x7f2a3409c150 con 0x7f2a441010a0 2024-04-23T18:25:37.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:25:37.641 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:25:33.806672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:25:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 41 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 msgr2=0x7f2a2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 0x7f2a2c0a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2a440781d0 tx=0x7f2a38005f90 comp rx=0 tx=0).stop 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 msgr2=0x7f2a441986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400d950 tx=0x7f2a3400de10 comp rx=0 tx=0).stop 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 shutdown_connections 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a44106ed0 0x7f2a4419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44101a40 0x7f2a44198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2a2c0a3d20 0x7f2a2c0a61e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 --2- 172.21.15.32:0/1128611917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a441010a0 0x7f2a441986e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 >> 172.21.15.32:0/1128611917 conn(0x7f2a44077c40 msgr2=0x7f2a44069410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 shutdown_connections 2024-04-23T18:25:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.650+0000 7f2a4acfd640 1 -- 172.21.15.32:0/1128611917 wait complete. 2024-04-23T18:25:37.655 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 6f37624f --release --await' 2024-04-23T18:25:37.655 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 6f37624f --release --await 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- 172.21.15.32:0/1662777676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 msgr2=0x7feb3c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 --2- 172.21.15.32:0/1662777676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c10e9a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7feb34009ca0 tx=0x7feb3402a950 comp rx=0 tx=0).stop 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- 172.21.15.32:0/1662777676 shutdown_connections 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 --2- 172.21.15.32:0/1662777676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c10e9a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 --2- 172.21.15.32:0/1662777676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c103ae0 0x7feb3c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 --2- 172.21.15.32:0/1662777676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c1031c0 0x7feb3c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- 172.21.15.32:0/1662777676 >> 172.21.15.32:0/1662777676 conn(0x7feb3c078000 msgr2=0x7feb3c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- 172.21.15.32:0/1662777676 shutdown_connections 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- 172.21.15.32:0/1662777676 wait complete. 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 Processor -- start 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.772+0000 7feb43f68640 1 -- start start 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 0x7feb3c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c103ae0 0x7feb3c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1121e0 con 0x7feb3c10c500 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c112060 con 0x7feb3c103ae0 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c112360 con 0x7feb3c1031c0 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45376/0 (socket says 172.21.15.32:45376) 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 -- 172.21.15.32:0/2745791361 learned_addr learned my addr 172.21.15.32:0/2745791361 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb41503640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c103ae0 0x7feb3c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb41d04640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 0x7feb3c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 -- 172.21.15.32:0/2745791361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 msgr2=0x7feb3c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 0x7feb3c100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 -- 172.21.15.32:0/2745791361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c103ae0 msgr2=0x7feb3c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c103ae0 0x7feb3c101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 -- 172.21.15.32:0/2745791361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c1a6c40 con 0x7feb3c10c500 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb41d04640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 0x7feb3c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb42505640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feb34009dc0 tx=0x7feb34003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb32ffd640 1 -- 172.21.15.32:0/2745791361 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34034040 con 0x7feb3c10c500 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb32ffd640 1 -- 172.21.15.32:0/2745791361 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb34030850 con 0x7feb3c10c500 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3c1a6ed0 con 0x7feb3c10c500 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb32ffd640 1 -- 172.21.15.32:0/2745791361 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34004680 con 0x7feb3c10c500 2024-04-23T18:25:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.773+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3c1a7390 con 0x7feb3c10c500 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb32ffd640 1 -- 172.21.15.32:0/2745791361 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7feb34037020 con 0x7feb3c10c500 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 Processor -- start 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 -- start start 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a2233db0 0x55a3a223df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a223e480 0x55a3a2240870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a3a21986d0 con 0x55a3a2233490 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a3a21351a0 con 0x55a3a2233db0 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.776+0000 7feb46c80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a3a2194690 con 0x55a3a223e480 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45380/0 (socket says 172.21.15.32:45380) 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 -- 172.21.15.32:0/1939422195 learned_addr learned my addr 172.21.15.32:0/1939422195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb32ffd640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 0x7feb280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41d04640 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a2233db0 0x55a3a223df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a223e480 0x55a3a2240870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb42505640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 0x7feb280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 -- 172.21.15.32:0/1939422195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a223e480 msgr2=0x55a3a2240870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a223e480 0x55a3a2240870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 -- 172.21.15.32:0/1939422195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a2233db0 msgr2=0x55a3a223df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a2233db0 0x55a3a223df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 -- 172.21.15.32:0/1939422195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a213f5b0 con 0x55a3a2233490 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb32ffd640 1 -- 172.21.15.32:0/2745791361 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7feb340f1560 con 0x7feb3c10c500 2024-04-23T18:25:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb41503640 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00bb20 tx=0x7feb2c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b808c8ae313e17e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb27fff640 1 -- 172.21.15.32:0/1939422195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c005020 con 0x55a3a2233490 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb42505640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 0x7feb280a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb340bd5c0 tx=0x7feb3402fe30 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb27fff640 1 -- 172.21.15.32:0/1939422195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c005670 con 0x55a3a2233490 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.777+0000 7feb27fff640 1 -- 172.21.15.32:0/1939422195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c010460 con 0x55a3a2233490 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- 172.21.15.32:0/1939422195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 msgr2=0x55a3a2233870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00bb20 tx=0x7feb2c0041d0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- 172.21.15.32:0/1939422195 shutdown_connections 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a223e480 0x55a3a2240870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a2233db0 0x55a3a223df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 --2- 172.21.15.32:0/1939422195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a2233870 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- 172.21.15.32:0/1939422195 >> 172.21.15.32:0/1939422195 conn(0x55a3a222f7b0 msgr2=0x55a3a222fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- 172.21.15.32:0/1939422195 shutdown_connections 2024-04-23T18:25:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- 172.21.15.32:0/1939422195 wait complete. 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.778+0000 7feb46c80740 1 -- start start 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 Processor -- start 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a223b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a3a22412b0 con 0x55a3a2233490 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a3a2132b20 con 0x55a3a223e480 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a3a2241430 con 0x55a3a2233db0 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb46c80740 10 client.0 _ll_register_callbacks cb 0x55a3a2132530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45384/0 (socket says 172.21.15.32:45384) 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 -- 172.21.15.32:0/4276803086 learned_addr learned my addr 172.21.15.32:0/4276803086 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41d04640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a223b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 -- 172.21.15.32:0/4276803086 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 msgr2=0x55a3a223b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a223b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 -- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 msgr2=0x55a3a223b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.779+0000 7feb41503640 1 -- 172.21.15.32:0/4276803086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a1eaa920 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb41503640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00b5f0 tx=0x7feb2c003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43f934c02c4343b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c004ad0 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7feb2c005000 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2c005670 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7feb2c005200 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 10 client.16755 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a3a1f76d30 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c005670 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7feb2c020450 con 0x55a3a2233490 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 10 client.16755 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 10 client.16755 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:25:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 20 client.16755 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a224e4f0 0x55a3a22508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 4 client.16755 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.780+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3a2156500 con 0x55a3a224e4f0 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.781+0000 7feb41d04640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a224e4f0 0x55a3a22508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.781+0000 7feb41d04640 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a224e4f0 0x55a3a22508e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a16be9ba61ce7088 server_cookie=6f9e6aba546a0426 in_seq=0 out_seq=0 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.781+0000 7feb257fa640 10 client.16755 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb257fa640 1 -- 172.21.15.32:0/4276803086 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3a2156500 con 0x55a3a224e4f0 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb257fa640 10 client.16755 handle_command_reply: tid=0 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 client.16755 shutdown 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 msgr2=0x55a3a223dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00b5f0 tx=0x7feb2c003cf0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 shutdown_connections 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a224e4f0 0x55a3a22508e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a223b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb257fa640 0 client.16755 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 --2- 172.21.15.32:0/4276803086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a223dec0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 >> 172.21.15.32:0/4276803086 conn(0x55a3a222f7b0 msgr2=0x55a3a22354b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 shutdown_connections 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 1 -- 172.21.15.32:0/4276803086 wait complete. 2024-04-23T18:25:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.782+0000 7feb46c80740 20 client.16755 trim_cache size 0 max 16384 2024-04-23T18:25:37.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 Processor -- start 2024-04-23T18:25:37.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 -- start start 2024-04-23T18:25:37.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233490 0x55a3a22366b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1b25e0 con 0x55a3a2233db0 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1afe80 con 0x55a3a223e480 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb46c80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1adcf0 con 0x55a3a2233490 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45394/0 (socket says 172.21.15.32:45394) 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 -- 172.21.15.32:0/3114031369 learned_addr learned my addr 172.21.15.32:0/3114031369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41503640 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233490 0x55a3a22366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 -- 172.21.15.32:0/3114031369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233490 msgr2=0x55a3a22366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233490 0x55a3a22366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 -- 172.21.15.32:0/3114031369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 msgr2=0x55a3a223d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 -- 172.21.15.32:0/3114031369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a2233290 con 0x55a3a2233db0 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb41d04640 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feb3800bb50 tx=0x7feb3800b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e726a881996ab92c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb277fe640 1 -- 172.21.15.32:0/3114031369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3801bbc0 con 0x55a3a2233db0 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb277fe640 1 -- 172.21.15.32:0/3114031369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb38020070 con 0x55a3a2233db0 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.800+0000 7feb277fe640 1 -- 172.21.15.32:0/3114031369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3801b610 con 0x55a3a2233db0 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- 172.21.15.32:0/3114031369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 msgr2=0x55a3a222e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feb3800bb50 tx=0x7feb3800b5a0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- 172.21.15.32:0/3114031369 shutdown_connections 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a223d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233db0 0x55a3a222e850 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 --2- 172.21.15.32:0/3114031369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233490 0x55a3a22366b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- 172.21.15.32:0/3114031369 >> 172.21.15.32:0/3114031369 conn(0x55a3a222f7b0 msgr2=0x55a3a222e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- 172.21.15.32:0/3114031369 shutdown_connections 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- 172.21.15.32:0/3114031369 wait complete. 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.801+0000 7feb46c80740 1 -- start start 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 Processor -- start 2024-04-23T18:25:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a2246bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a2237420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1ae340 con 0x55a3a2233490 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1af7b0 con 0x55a3a223e480 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c1aea10 con 0x55a3a2233db0 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb46c80740 10 client.0 _ll_register_callbacks cb 0x7feb2c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45398/0 (socket says 172.21.15.32:45398) 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 -- 172.21.15.32:0/832095727 learned_addr learned my addr 172.21.15.32:0/832095727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41d04640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a2246bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a2237420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 -- 172.21.15.32:0/832095727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 msgr2=0x55a3a2246bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a2246bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 -- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 msgr2=0x55a3a2237420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a2237420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 -- 172.21.15.32:0/832095727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a2237a90 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb41503640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feb2c020850 tx=0x7feb2c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5019ffbc7594bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.802+0000 7feb257fa640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c020650 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7feb2c020a40 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb38020070 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7feb2c020c40 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 10 client.16761 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a3a2238dd0 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb38020070 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7feb2c03e020 con 0x55a3a2233490 2024-04-23T18:25:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 10 client.16761 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 10 client.16761 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 20 client.16761 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a22b5940 0x55a3a22b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 4 client.16761 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6f37624f", "release": true, "await": true}] 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6f37624f", "release": true, "await": true}) -- 0x55a3a21a60f0 con 0x55a3a22b5940 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.803+0000 7feb41d04640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a22b5940 0x55a3a22b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.804+0000 7feb257fa640 10 client.16761 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.804+0000 7feb41d04640 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a22b5940 0x55a3a22b7d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab7d7537c54205a3 server_cookie=4e717177b9dc4a1a in_seq=0 out_seq=0 2024-04-23T18:25:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb257fa640 1 -- 172.21.15.32:0/832095727 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a3a21a60f0 con 0x55a3a22b5940 2024-04-23T18:25:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb257fa640 10 client.16761 handle_command_reply: tid=0 2024-04-23T18:25:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 client.16761 shutdown 2024-04-23T18:25:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 msgr2=0x55a3a22466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feb2c020850 tx=0x7feb2c0041d0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 shutdown_connections 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a3a22b5940 0x55a3a22b7d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a223e480 0x55a3a2237420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a3a2233db0 0x55a3a2246bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 --2- 172.21.15.32:0/832095727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a3a2233490 0x55a3a22466b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb257fa640 0 client.16761 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 >> 172.21.15.32:0/832095727 conn(0x55a3a222f7b0 msgr2=0x55a3a222e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 shutdown_connections 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 1 -- 172.21.15.32:0/832095727 wait complete. 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.811+0000 7feb46c80740 20 client.16761 trim_cache size 0 max 16384 2024-04-23T18:25:37.806 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 41, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 36, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "6f37624f": { 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 36, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:25:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 72.0, 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:25:37.808 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:25:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 msgr2=0x7feb280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 0x7feb280a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb340bd5c0 tx=0x7feb3402fe30 comp rx=0 tx=0).stop 2024-04-23T18:25:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 msgr2=0x7feb3c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feb34009dc0 tx=0x7feb34003f10 comp rx=0 tx=0).stop 2024-04-23T18:25:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 shutdown_connections 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feb280a3d20 0x7feb280a61e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb3c10c500 0x7feb3c101890 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb3c103ae0 0x7feb3c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 --2- 172.21.15.32:0/2745791361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb3c1031c0 0x7feb3c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 >> 172.21.15.32:0/2745791361 conn(0x7feb3c078000 msgr2=0x7feb3c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 shutdown_connections 2024-04-23T18:25:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:25:37.812+0000 7feb43f68640 1 -- 172.21.15.32:0/2745791361 wait complete. 2024-04-23T18:25:37.817 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6f37624f', seconds elapsed: 13.7 2024-04-23T18:25:38.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:37 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1128611917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:38.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:37 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1128611917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:38.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:37 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1128611917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 610.0 s, 539.7 tps, lat 513.496 ms stddev 2423.818 2024-04-23T18:25:39.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:38 smithi032 ceph-mon[29221]: pgmap v327: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:25:39.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:38 smithi040 ceph-mon[32626]: pgmap v327: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:25:39.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:38 smithi064 ceph-mon[33062]: pgmap v327: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-23T18:25:41.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:40 smithi032 ceph-mon[29221]: pgmap v328: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-23T18:25:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:40 smithi040 ceph-mon[32626]: pgmap v328: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-23T18:25:41.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:40 smithi064 ceph-mon[33062]: pgmap v328: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-23T18:25:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:42 smithi040 ceph-mon[32626]: pgmap v329: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 982 KiB/s wr, 77 op/s 2024-04-23T18:25:43.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:42 smithi064 ceph-mon[33062]: pgmap v329: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 982 KiB/s wr, 77 op/s 2024-04-23T18:25:43.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:42 smithi032 ceph-mon[29221]: pgmap v329: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 982 KiB/s wr, 77 op/s 2024-04-23T18:25:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 615.0 s, 2661.4 tps, lat 37.223 ms stddev 41.098 2024-04-23T18:25:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.312+0000 7f8315ec2640 1 -- 172.21.15.32:0/3569948901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310106170 msgr2=0x7f831010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.312+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3569948901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310106170 0x7f831010dd40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8308009ca0 tx=0x7f830802a8b0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.312+0000 7f8315ec2640 1 -- 172.21.15.32:0/3569948901 shutdown_connections 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.312+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3569948901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310106170 0x7f831010dd40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.313+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3569948901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310101a50 0x7f8310105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.313+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3569948901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83101010a0 0x7f8310101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.313+0000 7f8315ec2640 1 -- 172.21.15.32:0/3569948901 >> 172.21.15.32:0/3569948901 conn(0x7f8310077f70 msgr2=0x7f8310078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.313+0000 7f8315ec2640 1 -- 172.21.15.32:0/3569948901 shutdown_connections 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.313+0000 7f8315ec2640 1 -- 172.21.15.32:0/3569948901 wait complete. 2024-04-23T18:25:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.314+0000 7f8315ec2640 1 Processor -- start 2024-04-23T18:25:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 -- start start 2024-04-23T18:25:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8310106170 0x7f8310070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 0x7f83100715a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f831006c270 con 0x7f83101010a0 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f831006c0f0 con 0x7f8310106170 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f8315ec2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f831006c3f0 con 0x7f8310071110 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f830f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f830f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45412/0 (socket says 172.21.15.32:45412) 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f830f7fe640 1 -- 172.21.15.32:0/3301731393 learned_addr learned my addr 172.21.15.32:0/3301731393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f830effd640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8310106170 0x7f8310070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.315+0000 7f830ffff640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 0x7f83100715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 -- 172.21.15.32:0/3301731393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 msgr2=0x7f83100715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 0x7f83100715a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 -- 172.21.15.32:0/3301731393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8310106170 msgr2=0x7f8310070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8310106170 0x7f8310070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 -- 172.21.15.32:0/3301731393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83101af780 con 0x7f83101010a0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830ffff640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 0x7f83100715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.316+0000 7f830f7fe640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f830400ea60 tx=0x7f830400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.317+0000 7f830cff9640 1 -- 172.21.15.32:0/3301731393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830400cde0 con 0x7f83101010a0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.317+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83101afa70 con 0x7f83101010a0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.317+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83101affc0 con 0x7f83101010a0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.317+0000 7f830cff9640 1 -- 172.21.15.32:0/3301731393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83040035f0 con 0x7f83101010a0 2024-04-23T18:25:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.317+0000 7f830cff9640 1 -- 172.21.15.32:0/3301731393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304010690 con 0x7f83101010a0 2024-04-23T18:25:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.320+0000 7f8318bd3740 1 Processor -- start 2024-04-23T18:25:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 -- start start 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c7ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6bd6970 con 0x55e9e6c71f40 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6b734a0 con 0x55e9e6c7c800 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f8318bd3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6bd2930 con 0x55e9e6c71620 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f830ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f830ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:37038/0 (socket says 172.21.15.32:37038) 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f830ffff640 1 -- 172.21.15.32:0/2823209623 learned_addr learned my addr 172.21.15.32:0/2823209623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.321+0000 7f830effd640 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c7ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 -- 172.21.15.32:0/2823209623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 msgr2=0x55e9e6c7ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c7ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 -- 172.21.15.32:0/2823209623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 msgr2=0x55e9e6c7c2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:25:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7c2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 -- 172.21.15.32:0/2823209623 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9e68e9210 con 0x55e9e6c71620 2024-04-23T18:25:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830cff9640 1 -- 172.21.15.32:0/3301731393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8304004000 con 0x7f83101010a0 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.322+0000 7f830ffff640 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8308035000 tx=0x7f8308030f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d602d672d0924241 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.323+0000 7f82f5ffb640 1 -- 172.21.15.32:0/2823209623 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308030620 con 0x55e9e6c71620 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.323+0000 7f82f5ffb640 1 -- 172.21.15.32:0/2823209623 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8308036040 con 0x55e9e6c71620 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.323+0000 7f82f5ffb640 1 -- 172.21.15.32:0/2823209623 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83080043a0 con 0x55e9e6c71620 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.323+0000 7f830cff9640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 0x7f82f80aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.323+0000 7f830cff9640 1 -- 172.21.15.32:0/3301731393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f83040d08b0 con 0x7f83101010a0 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 -- 172.21.15.32:0/2823209623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 msgr2=0x55e9e6c71a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8308035000 tx=0x7f8308030f00 comp rx=0 tx=0).stop 2024-04-23T18:25:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f830f7fe640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 0x7f82f80aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 -- 172.21.15.32:0/2823209623 shutdown_connections 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c7ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2823209623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71620 0x55e9e6c71a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.324+0000 7f8318bd3740 1 -- 172.21.15.32:0/2823209623 >> 172.21.15.32:0/2823209623 conn(0x55e9e6c6db30 msgr2=0x55e9e6c6df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.325+0000 7f830f7fe640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 0x7f82f80aeb70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55e9e6c72bc0 tx=0x7f8304009490 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.325+0000 7f8318bd3740 1 -- 172.21.15.32:0/2823209623 shutdown_connections 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.325+0000 7f8318bd3740 1 -- 172.21.15.32:0/2823209623 wait complete. 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.325+0000 7f8318bd3740 1 -- start start 2024-04-23T18:25:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 Processor -- start 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c7c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 0x55e9e6c79940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6c7f3d0 con 0x55e9e6c71f40 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6b710b0 con 0x55e9e6c71620 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e9e6c7f550 con 0x55e9e6c7c800 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f8318bd3740 10 client.0 _ll_register_callbacks cb 0x55e9e6b5ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f830effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f830effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45428/0 (socket says 172.21.15.32:45428) 2024-04-23T18:25:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.326+0000 7f830effd640 1 -- 172.21.15.32:0/1597232685 learned_addr learned my addr 172.21.15.32:0/1597232685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 -- 172.21.15.32:0/1597232685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 msgr2=0x55e9e6c79940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830ffff640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 0x55e9e6c79940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 0x55e9e6c79940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 -- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 msgr2=0x55e9e6c7c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c7c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 -- 172.21.15.32:0/1597232685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9e69b5620 con 0x55e9e6c71f40 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830ffff640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 0x55e9e6c79940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f830effd640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f83000126d0 tx=0x7f8300012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee849997ccf0f4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f82eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300004030 con 0x55e9e6c71f40 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f83000041d0 con 0x55e9e6c71f40 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8308036040 con 0x55e9e6c71f40 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.327+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f83000043d0 con 0x55e9e6c71f40 2024-04-23T18:25:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 10 client.16770 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e9e6c8b3e0 con 0x55e9e6c71f40 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8308036040 con 0x55e9e6c71f40 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f8300023020 con 0x55e9e6c71f40 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 10 client.16770 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 10 client.16770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 20 client.16770 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6c8c8c0 0x55e9e6c8ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 4 client.16770 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9e6be4390 con 0x55e9e6c8c8c0 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.328+0000 7f830ffff640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6c8c8c0 0x55e9e6c8ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.329+0000 7f82eeffd640 10 client.16770 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.329+0000 7f830ffff640 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6c8c8c0 0x55e9e6c8ed40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fda5a05f1303f1 server_cookie=1f86f735ad84799a in_seq=0 out_seq=0 2024-04-23T18:25:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.329+0000 7f82eeffd640 1 -- 172.21.15.32:0/1597232685 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9e6be4390 con 0x55e9e6c8c8c0 2024-04-23T18:25:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.329+0000 7f82eeffd640 10 client.16770 handle_command_reply: tid=0 2024-04-23T18:25:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.329+0000 7f8318bd3740 1 client.16770 shutdown 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 msgr2=0x55e9e6c79400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f83000126d0 tx=0x7f8300012b90 comp rx=0 tx=0).stop 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 shutdown_connections 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6c8c8c0 0x55e9e6c8ed40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c7c800 0x55e9e6c79940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71f40 0x55e9e6c79400 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 --2- 172.21.15.32:0/1597232685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c7c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 >> 172.21.15.32:0/1597232685 conn(0x55e9e6c6db30 msgr2=0x55e9e6c738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.330+0000 7f82eeffd640 0 client.16770 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.331+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 shutdown_connections 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.331+0000 7f8318bd3740 1 -- 172.21.15.32:0/1597232685 wait complete. 2024-04-23T18:25:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.331+0000 7f8318bd3740 20 client.16770 trim_cache size 0 max 16384 2024-04-23T18:25:44.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.355+0000 7f8318bd3740 1 Processor -- start 2024-04-23T18:25:44.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.355+0000 7f8318bd3740 1 -- start start 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.355+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.355+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c7c800 0x55e9e6c783e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f8318bd3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc010fd0 con 0x55e9e6c7c800 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f8318bd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc00e870 con 0x55e9e6c71620 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f8318bd3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc00c6e0 con 0x55e9e6c71f40 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:37062/0 (socket says 172.21.15.32:37062) 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 -- 172.21.15.32:0/2701973533 learned_addr learned my addr 172.21.15.32:0/2701973533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830ffff640 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c7c800 0x55e9e6c783e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 -- 172.21.15.32:0/2701973533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 msgr2=0x55e9e6c882f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c882f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 -- 172.21.15.32:0/2701973533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c7c800 msgr2=0x55e9e6c783e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c7c800 0x55e9e6c783e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 -- 172.21.15.32:0/2701973533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9e6c72530 con 0x55e9e6c71f40 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830ffff640 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c882f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.356+0000 7f830effd640 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8300003c50 tx=0x7f830000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c656d529c46025 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.357+0000 7f82f57fa640 1 -- 172.21.15.32:0/2701973533 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000035f0 con 0x55e9e6c71f40 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.357+0000 7f82f57fa640 1 -- 172.21.15.32:0/2701973533 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830002a070 con 0x55e9e6c71f40 2024-04-23T18:25:44.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.357+0000 7f82f57fa640 1 -- 172.21.15.32:0/2701973533 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830000fd80 con 0x55e9e6c71f40 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.357+0000 7f8318bd3740 1 -- 172.21.15.32:0/2701973533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 msgr2=0x55e9e6c74af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8300003c50 tx=0x7f830000f790 comp rx=0 tx=0).stop 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 -- 172.21.15.32:0/2701973533 shutdown_connections 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c7c800 0x55e9e6c783e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c74af0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 --2- 172.21.15.32:0/2701973533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c71620 0x55e9e6c882f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 -- 172.21.15.32:0/2701973533 >> 172.21.15.32:0/2701973533 conn(0x55e9e6c6db30 msgr2=0x55e9e6c77cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 -- 172.21.15.32:0/2701973533 shutdown_connections 2024-04-23T18:25:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 -- 172.21.15.32:0/2701973533 wait complete. 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 -- start start 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.358+0000 7f8318bd3740 1 Processor -- start 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 0x55e9e6c84a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c80300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc00cd30 con 0x55e9e6c71620 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc00e1a0 con 0x55e9e6c7c800 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc00d400 con 0x55e9e6c71f40 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f8318bd3740 10 client.0 _ll_register_callbacks cb 0x7f8300009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 0x55e9e6c84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 0x55e9e6c84a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45446/0 (socket says 172.21.15.32:45446) 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830ffff640 1 -- 172.21.15.32:0/3598495383 learned_addr learned my addr 172.21.15.32:0/3598495383 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830ffff640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c80300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 -- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 msgr2=0x55e9e6c80300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c80300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 -- 172.21.15.32:0/3598495383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 msgr2=0x55e9e6c84a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 0x55e9e6c84a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830effd640 1 -- 172.21.15.32:0/3598495383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9e6c80940 con 0x55e9e6c71f40 2024-04-23T18:25:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.359+0000 7f830ffff640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c80300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.360+0000 7f830effd640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7fdc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55e9e6c7f3c0 tx=0x7f830000f830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d2a57f2f9401afe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.360+0000 7f82eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300009e90 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f8300003e90 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f830002a070 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f830000fe90 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f8318bd3740 10 client.26338 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e9e6cf2d10 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.361+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830002a070 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f8300023070 con 0x55e9e6c71f40 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 10 client.26338 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 10 client.26338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 20 client.26338 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6cf3350 0x55e9e6cf5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 4 client.26338 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9e6bde200 con 0x55e9e6cf3350 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.362+0000 7f830ffff640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6cf3350 0x55e9e6cf5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.363+0000 7f830ffff640 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6cf3350 0x55e9e6cf5770 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606423ca1b446182 server_cookie=67482220c4523114 in_seq=0 out_seq=0 2024-04-23T18:25:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.363+0000 7f82eeffd640 10 client.26338 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f82eeffd640 1 -- 172.21.15.32:0/3598495383 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9e6bde200 con 0x55e9e6cf3350 2024-04-23T18:25:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f82eeffd640 10 client.26338 handle_command_reply: tid=0 2024-04-23T18:25:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 client.26338 shutdown 2024-04-23T18:25:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 msgr2=0x55e9e6c7fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7fdc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55e9e6c7f3c0 tx=0x7f830000f830 comp rx=0 tx=0).stop 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 shutdown_connections 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e9e6cf3350 0x55e9e6cf5770 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9e6c7c800 0x55e9e6c80300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e9e6c71f40 0x55e9e6c7fdc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 --2- 172.21.15.32:0/3598495383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e9e6c71620 0x55e9e6c84a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 >> 172.21.15.32:0/3598495383 conn(0x55e9e6c6db30 msgr2=0x55e9e6c6c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.392+0000 7f82eeffd640 0 client.26338 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.393+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 shutdown_connections 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.393+0000 7f8318bd3740 1 -- 172.21.15.32:0/3598495383 wait complete. 2024-04-23T18:25:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.393+0000 7f8318bd3740 20 client.26338 trim_cache size 0 max 16384 2024-04-23T18:25:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:25:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:25:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:25:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 msgr2=0x7f82f80aeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 0x7f82f80aeb70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55e9e6c72bc0 tx=0x7f8304009490 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 msgr2=0x7f8310073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f830400ea60 tx=0x7f830400ef20 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 shutdown_connections 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8310071110 0x7f83100715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8310106170 0x7f8310070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f82f80ac6b0 0x7f82f80aeb70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 --2- 172.21.15.32:0/3301731393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83101010a0 0x7f8310073b30 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 >> 172.21.15.32:0/3301731393 conn(0x7f8310077f70 msgr2=0x7f83100fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 shutdown_connections 2024-04-23T18:25:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:44.394+0000 7f8315ec2640 1 -- 172.21.15.32:0/3301731393 wait complete. 2024-04-23T18:25:45.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:44 smithi040 ceph-mon[32626]: pgmap v330: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-23T18:25:45.173 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:44 smithi064 ceph-mon[33062]: pgmap v330: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-23T18:25:45.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:44 smithi032 ceph-mon[29221]: pgmap v330: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-23T18:25:47.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:46 smithi032 ceph-mon[29221]: pgmap v331: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 5.3 MiB/s wr, 364 op/s 2024-04-23T18:25:47.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:46 smithi040 ceph-mon[32626]: pgmap v331: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 5.3 MiB/s wr, 364 op/s 2024-04-23T18:25:47.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:46 smithi064 ceph-mon[33062]: pgmap v331: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 5.3 MiB/s wr, 364 op/s 2024-04-23T18:25:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 620.0 s, 2466.2 tps, lat 40.934 ms stddev 45.663 2024-04-23T18:25:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:48 smithi040 ceph-mon[32626]: pgmap v332: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-04-23T18:25:49.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:48 smithi064 ceph-mon[33062]: pgmap v332: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-04-23T18:25:49.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:48 smithi032 ceph-mon[29221]: pgmap v332: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-04-23T18:25:51.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:50 smithi040 ceph-mon[32626]: pgmap v333: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 10 MiB/s wr, 685 op/s 2024-04-23T18:25:51.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:50 smithi064 ceph-mon[33062]: pgmap v333: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 10 MiB/s wr, 685 op/s 2024-04-23T18:25:51.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:50 smithi032 ceph-mon[29221]: pgmap v333: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 10 MiB/s wr, 685 op/s 2024-04-23T18:25:53.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:52 smithi040 ceph-mon[32626]: pgmap v334: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 617 op/s 2024-04-23T18:25:53.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:52 smithi064 ceph-mon[33062]: pgmap v334: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 617 op/s 2024-04-23T18:25:53.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:52 smithi032 ceph-mon[29221]: pgmap v334: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 617 op/s 2024-04-23T18:25:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 625.0 s, 2422.2 tps, lat 41.271 ms stddev 44.866 2024-04-23T18:25:55.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:54 smithi040 ceph-mon[32626]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 622 op/s 2024-04-23T18:25:55.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:54 smithi064 ceph-mon[33062]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 622 op/s 2024-04-23T18:25:55.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:54 smithi032 ceph-mon[29221]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.4 MiB/s wr, 622 op/s 2024-04-23T18:25:57.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:56 smithi040 ceph-mon[32626]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 886 op/s 2024-04-23T18:25:57.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:56 smithi064 ceph-mon[33062]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 886 op/s 2024-04-23T18:25:57.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:56 smithi032 ceph-mon[29221]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 886 op/s 2024-04-23T18:25:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.550+0000 7f693b1e2640 1 -- 172.21.15.32:0/2475613302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f693411b730 msgr2=0x7f693411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.550+0000 7f693b1e2640 1 --2- 172.21.15.32:0/2475613302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f693411b730 0x7f693411db20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f692c009ca0 tx=0x7f692c02a8f0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.557+0000 7f693b1e2640 1 -- 172.21.15.32:0/2475613302 shutdown_connections 2024-04-23T18:25:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.557+0000 7f693b1e2640 1 --2- 172.21.15.32:0/2475613302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f693411b730 0x7f693411db20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.557+0000 7f693b1e2640 1 --2- 172.21.15.32:0/2475613302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f693410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.557+0000 7f693b1e2640 1 --2- 172.21.15.32:0/2475613302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69341106f0 0x7f693410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.557+0000 7f693b1e2640 1 -- 172.21.15.32:0/2475613302 >> 172.21.15.32:0/2475613302 conn(0x7f693406d030 msgr2=0x7f693406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.556 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.563+0000 7f693b1e2640 1 -- 172.21.15.32:0/2475613302 shutdown_connections 2024-04-23T18:25:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.563+0000 7f693b1e2640 1 -- 172.21.15.32:0/2475613302 wait complete. 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.564+0000 7f693b1e2640 1 Processor -- start 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.564+0000 7f693b1e2640 1 -- start start 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.564+0000 7f693b1e2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f69341a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f69341a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693b1e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69341106f0 0x7f69341a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f69341a7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56106/0 (socket says 172.21.15.32:56106) 2024-04-23T18:25:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693b1e2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 0x7f69341abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693b1e2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6934121020 con 0x7f693411b730 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693b1e2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6934120ea0 con 0x7f693410e7d0 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693b1e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69341211a0 con 0x7f69341106f0 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f693977f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 0x7f69341abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.565+0000 7f6938f7e640 1 -- 172.21.15.32:0/379695415 learned_addr learned my addr 172.21.15.32:0/379695415 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.566+0000 7f693977f640 1 -- 172.21.15.32:0/379695415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69341106f0 msgr2=0x7f69341a7a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.566+0000 7f693977f640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69341106f0 0x7f69341a7a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.566+0000 7f693977f640 1 -- 172.21.15.32:0/379695415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 msgr2=0x7f69341a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.566+0000 7f693977f640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f69341a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.566+0000 7f693977f640 1 -- 172.21.15.32:0/379695415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69341ac510 con 0x7f693411b730 2024-04-23T18:25:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.567+0000 7f693977f640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 0x7f69341abde0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f692c009bf0 tx=0x7f692c02f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.569+0000 7f6931ffb640 1 -- 172.21.15.32:0/379695415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c02fa50 con 0x7f693411b730 2024-04-23T18:25:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.569+0000 7f6931ffb640 1 -- 172.21.15.32:0/379695415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f692c02d070 con 0x7f693411b730 2024-04-23T18:25:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.569+0000 7f6931ffb640 1 -- 172.21.15.32:0/379695415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c0466e0 con 0x7f693411b730 2024-04-23T18:25:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.570+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69341ac7a0 con 0x7f693411b730 2024-04-23T18:25:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.571+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69341b4090 con 0x7f693411b730 2024-04-23T18:25:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.574+0000 7f6931ffb640 1 -- 172.21.15.32:0/379695415 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f692c0305f0 con 0x7f693411b730 2024-04-23T18:25:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.575+0000 7f6931ffb640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 0x7f691c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.576+0000 7f6931ffb640 1 -- 172.21.15.32:0/379695415 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f692c0f1c70 con 0x7f693411b730 2024-04-23T18:25:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.576+0000 7f6938f7e640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 0x7f691c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.577+0000 7f693dfd2740 1 Processor -- start 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.577+0000 7f693dfd2740 1 -- start start 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9620 0x55b1a74d9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b1a743e970 con 0x55b1a74d9620 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b1a73db4a0 con 0x55b1a74e4800 2024-04-23T18:25:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.578+0000 7f693dfd2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b1a743a930 con 0x55b1a74d9f40 2024-04-23T18:25:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f6938f7e640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 0x7f691c0a63e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6924005fd0 tx=0x7f692400f730 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56112/0 (socket says 172.21.15.32:56112) 2024-04-23T18:25:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 -- 172.21.15.32:0/3960377419 learned_addr learned my addr 172.21.15.32:0/3960377419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9620 0x55b1a74d9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 -- 172.21.15.32:0/3960377419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 msgr2=0x55b1a74e42c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 -- 172.21.15.32:0/3960377419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9620 msgr2=0x55b1a74d9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9620 0x55b1a74d9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.580+0000 7f693977f640 1 -- 172.21.15.32:0/3960377419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a7151210 con 0x55b1a74e4800 2024-04-23T18:25:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.581+0000 7f693977f640 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f692c0c6420 tx=0x7f692c0bd8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aa1d2a7419460f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.582+0000 7f691affd640 1 -- 172.21.15.32:0/3960377419 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c0bdce0 con 0x55b1a74e4800 2024-04-23T18:25:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.582+0000 7f691affd640 1 -- 172.21.15.32:0/3960377419 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f692c036020 con 0x55b1a74e4800 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.582+0000 7f691affd640 1 -- 172.21.15.32:0/3960377419 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c0308e0 con 0x55b1a74e4800 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- 172.21.15.32:0/3960377419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 msgr2=0x55b1a74e6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f692c0c6420 tx=0x7f692c0bd8e0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- 172.21.15.32:0/3960377419 shutdown_connections 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74e6bf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3960377419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9620 0x55b1a74d9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- 172.21.15.32:0/3960377419 >> 172.21.15.32:0/3960377419 conn(0x55b1a74d5b30 msgr2=0x55b1a74d5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- 172.21.15.32:0/3960377419 shutdown_connections 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- 172.21.15.32:0/3960377419 wait complete. 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.584+0000 7f693dfd2740 1 -- start start 2024-04-23T18:25:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 Processor -- start 2024-04-23T18:25:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74e4800 0x55b1a74e1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b1a74e71f0 con 0x55b1a74e4800 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b1a73d90b0 con 0x55b1a74d9620 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b1a74e7370 con 0x55b1a74d9f40 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56126/0 (socket says 172.21.15.32:56126) 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f6938f7e640 1 -- 172.21.15.32:0/3452603848 learned_addr learned my addr 172.21.15.32:0/3452603848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693977f640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74e4800 0x55b1a74e1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f6933fff640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.585+0000 7f693dfd2740 10 client.0 _ll_register_callbacks cb 0x55b1a73c6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 -- 172.21.15.32:0/3452603848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 msgr2=0x55b1a74e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 -- 172.21.15.32:0/3452603848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74e4800 msgr2=0x55b1a74e1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74e4800 0x55b1a74e1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 -- 172.21.15.32:0/3452603848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a721d620 con 0x55b1a74d9620 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6933fff640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6924012820 tx=0x7f6924004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95441164fac715c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6913fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924027630 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f692401e410 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f692c036020 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f69240277d0 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f693dfd2740 10 client.26321 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55b1a74f2f20 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.586+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f692c036020 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f692402e070 con 0x55b1a74d9620 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 10 client.26321 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 10 client.26321 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 20 client.26321 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a74f4450 0x55b1a74f6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 4 client.26321 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1a744c390 con 0x55b1a74f4450 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.587+0000 7f6933fff640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a74f4450 0x55b1a74f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.588+0000 7f6913fff640 10 client.26321 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.588+0000 7f6933fff640 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a74f4450 0x55b1a74f6840 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f227453c17cb81de server_cookie=5823c1dbc68afe9a in_seq=0 out_seq=0 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.588+0000 7f6913fff640 1 -- 172.21.15.32:0/3452603848 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1a744c390 con 0x55b1a74f4450 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.588+0000 7f6913fff640 10 client.26321 handle_command_reply: tid=0 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.588+0000 7f693dfd2740 1 client.26321 shutdown 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 msgr2=0x55b1a74e4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6924012820 tx=0x7f6924004730 comp rx=0 tx=0).stop 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 shutdown_connections 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a74f4450 0x55b1a74f6840 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74e4800 0x55b1a74e1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9f40 0x55b1a74e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3452603848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74e4210 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.589+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 >> 172.21.15.32:0/3452603848 conn(0x55b1a74d5b30 msgr2=0x55b1a74db8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.590+0000 7f6913fff640 0 client.26321 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.590+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 shutdown_connections 2024-04-23T18:25:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.590+0000 7f693dfd2740 1 -- 172.21.15.32:0/3452603848 wait complete. 2024-04-23T18:25:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.590+0000 7f693dfd2740 20 client.26321 trim_cache size 0 max 16384 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.609+0000 7f693dfd2740 1 Processor -- start 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 -- start start 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74e4800 0x55b1a74e3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69341bf1a0 con 0x55b1a74d9f40 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69341bca40 con 0x55b1a74d9620 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693dfd2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69341ba8b0 con 0x55b1a74e4800 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56134/0 (socket says 172.21.15.32:56134) 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 -- 172.21.15.32:0/3678715860 learned_addr learned my addr 172.21.15.32:0/3678715860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6933fff640 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f693977f640 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74e4800 0x55b1a74e3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 -- 172.21.15.32:0/3678715860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74e4800 msgr2=0x55b1a74e3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74e4800 0x55b1a74e3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 -- 172.21.15.32:0/3678715860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 msgr2=0x55b1a74e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 -- 172.21.15.32:0/3678715860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a74e1e80 con 0x55b1a74d9620 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.610+0000 7f6938f7e640 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f692400c010 tx=0x7f692400b590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ffb1165d87aee00 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f691a7fc640 1 -- 172.21.15.32:0/3678715860 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924034070 con 0x55b1a74d9620 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f691a7fc640 1 -- 172.21.15.32:0/3678715860 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6924027970 con 0x55b1a74d9620 2024-04-23T18:25:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f691a7fc640 1 -- 172.21.15.32:0/3678715860 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924005280 con 0x55b1a74d9620 2024-04-23T18:25:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 -- 172.21.15.32:0/3678715860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 msgr2=0x55b1a74effe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f692400c010 tx=0x7f692400b590 comp rx=0 tx=0).stop 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 -- 172.21.15.32:0/3678715860 shutdown_connections 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74e4800 0x55b1a74e3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 --2- 172.21.15.32:0/3678715860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74d9620 0x55b1a74effe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 -- 172.21.15.32:0/3678715860 >> 172.21.15.32:0/3678715860 conn(0x55b1a74d5b30 msgr2=0x55b1a74e0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.611+0000 7f693dfd2740 1 -- 172.21.15.32:0/3678715860 shutdown_connections 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 -- 172.21.15.32:0/3678715860 wait complete. 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 -- start start 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 Processor -- start 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 0x55b1a74ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69341baf00 con 0x55b1a74d9f40 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69341bc370 con 0x55b1a74e4800 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69341bb5d0 con 0x55b1a74d9620 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f693dfd2740 10 client.0 _ll_register_callbacks cb 0x7f6924012520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f6933fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f6933fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44656/0 (socket says 172.21.15.32:44656) 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f6933fff640 1 -- 172.21.15.32:0/1319491639 learned_addr learned my addr 172.21.15.32:0/1319491639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.612+0000 7f6938f7e640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 0x55b1a74ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f693977f640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 -- 172.21.15.32:0/1319491639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 msgr2=0x55b1a74ec4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 0x55b1a74ec4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 -- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 msgr2=0x55b1a74ddc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74ddc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 -- 172.21.15.32:0/1319491639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a74de430 con 0x55b1a74d9f40 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6938f7e640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 0x55b1a74ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6933fff640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x55b1a74dabc0 tx=0x7f6928009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26ecad045d215fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6913fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928025060 con 0x55b1a74d9f40 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7f6928017730 con 0x55b1a74d9f40 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6924027970 con 0x55b1a74d9f40 2024-04-23T18:25:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6928017930 con 0x55b1a74d9f40 2024-04-23T18:25:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f693dfd2740 10 client.16794 fetch_fsmap learned FSMap version 41 2024-04-23T18:25:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.613+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55b1a755a550 con 0x55b1a74d9f40 2024-04-23T18:25:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.614+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6924027970 con 0x55b1a74d9f40 2024-04-23T18:25:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.615+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f6928015650 con 0x55b1a74d9f40 2024-04-23T18:25:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 10 client.16794 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 10 client.16794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 20 client.16794 populate_metadata read hostname 'smithi032' 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a755ac70 0x55b1a755d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 4 client.16794 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1a7446200 con 0x55b1a755ac70 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f6938f7e640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a755ac70 0x55b1a755d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f6938f7e640 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a755ac70 0x55b1a755d060 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe2cc0805fa25fa server_cookie=54dbc8b6ff3f078a in_seq=0 out_seq=0 2024-04-23T18:25:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.616+0000 7f6913fff640 10 client.16794 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.637+0000 7f6913fff640 1 -- 172.21.15.32:0/1319491639 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1a7446200 con 0x55b1a755ac70 2024-04-23T18:25:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.637+0000 7f6913fff640 10 client.16794 handle_command_reply: tid=0 2024-04-23T18:25:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.637+0000 7f693dfd2740 1 client.16794 shutdown 2024-04-23T18:25:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 msgr2=0x55b1a74eca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x55b1a74dabc0 tx=0x7f6928009d00 comp rx=0 tx=0).stop 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 shutdown_connections 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55b1a755ac70 0x55b1a755d060 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b1a74e4800 0x55b1a74ddc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1a74d9f40 0x55b1a74eca30 secure :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0x55b1a74dabc0 tx=0x7f6928009d00 comp rx=0 tx=0).stop 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 --2- 172.21.15.32:0/1319491639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b1a74d9620 0x55b1a74ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 >> 172.21.15.32:0/1319491639 conn(0x55b1a74d5b30 msgr2=0x55b1a74dc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f6913fff640 0 client.16794 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 shutdown_connections 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.638+0000 7f693dfd2740 1 -- 172.21.15.32:0/1319491639 wait complete. 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.639+0000 7f693dfd2740 20 client.16794 trim_cache size 0 max 16384 2024-04-23T18:25:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:25:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:25:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:25:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:25:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 msgr2=0x7f691c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 0x7f691c0a63e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6924005fd0 tx=0x7f692400f730 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 msgr2=0x7f69341abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 0x7f69341abde0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f692c009bf0 tx=0x7f692c02f410 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 shutdown_connections 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f691c0a3f20 0x7f691c0a63e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f693411b730 0x7f69341abde0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69341106f0 0x7f69341a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 --2- 172.21.15.32:0/379695415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693410e7d0 0x7f69341a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 >> 172.21.15.32:0/379695415 conn(0x7f693406d030 msgr2=0x7f693411a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 shutdown_connections 2024-04-23T18:25:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:25:57.640+0000 7f693b1e2640 1 -- 172.21.15.32:0/379695415 wait complete. 2024-04-23T18:25:58.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 630.0 s, 2280.8 tps, lat 43.637 ms stddev 48.742 2024-04-23T18:25:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:25:58 smithi040 ceph-mon[32626]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.4 MiB/s wr, 605 op/s 2024-04-23T18:25:59.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:25:58 smithi032 ceph-mon[29221]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.4 MiB/s wr, 605 op/s 2024-04-23T18:25:59.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:25:58 smithi064 ceph-mon[33062]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.4 MiB/s wr, 605 op/s 2024-04-23T18:26:00.104 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:26:00.104 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:26:00.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 -- 172.21.15.32:0/2215616497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858102b50 msgr2=0x7fe858106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/2215616497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858102b50 0x7fe858106b90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe848009900 tx=0x7fe84802a410 comp rx=0 tx=0).stop 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 -- 172.21.15.32:0/2215616497 shutdown_connections 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/2215616497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 0x7fe85810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/2215616497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858102b50 0x7fe858106b90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/2215616497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe858102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 -- 172.21.15.32:0/2215616497 >> 172.21.15.32:0/2215616497 conn(0x7fe8580fda40 msgr2=0x7fe8580ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 -- 172.21.15.32:0/2215616497 shutdown_connections 2024-04-23T18:26:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.259+0000 7fe85fdd8640 1 -- 172.21.15.32:0/2215616497 wait complete. 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 Processor -- start 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 -- start start 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe85819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 0x7fe85819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe858112190 con 0x7fe858198590 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe858112010 con 0x7fe8581021a0 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85fdd8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe858112310 con 0x7fe858107270 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44680/0 (socket says 172.21.15.32:44680) 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85db74640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe85819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 -- 172.21.15.32:0/664437869 learned_addr learned my addr 172.21.15.32:0/664437869 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85d373640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 0x7fe85819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 -- 172.21.15.32:0/664437869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 msgr2=0x7fe85819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 0x7fe85819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 -- 172.21.15.32:0/664437869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 msgr2=0x7fe85819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.260+0000 7fe85e375640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe85819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe85e375640 1 -- 172.21.15.32:0/664437869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8581990b0 con 0x7fe858198590 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe85db74640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe85819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe85e375640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe85000cad0 tx=0x7fe85000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85000ad60 con 0x7fe858198590 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8581993a0 con 0x7fe858198590 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe850005020 con 0x7fe858198590 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85000f6b0 con 0x7fe858198590 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.261+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8581a55f0 con 0x7fe858198590 2024-04-23T18:26:00.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.262+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe858069250 con 0x7fe858198590 2024-04-23T18:26:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.263+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe8500051d0 con 0x7fe858198590 2024-04-23T18:26:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.264+0000 7fe84effd640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 0x7fe8440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.264+0000 7fe85db74640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 0x7fe8440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.264+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7fe8500d0ab0 con 0x7fe858198590 2024-04-23T18:26:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.264+0000 7fe85db74640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 0x7fe8440a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe85400afb0 tx=0x7fe85400a2d0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.268+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe850014030 con 0x7fe858198590 2024-04-23T18:26:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.640+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe85810a4a0 con 0x7fe858198590 2024-04-23T18:26:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.641+0000 7fe84effd640 1 -- 172.21.15.32:0/664437869 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11852 (secure 0 0 0) 0x7fe85009d190 con 0x7fe858198590 2024-04-23T18:26:00.634 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:26:00.634 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:25:33.806672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:26:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 41 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 msgr2=0x7fe8440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 0x7fe8440a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe85400afb0 tx=0x7fe85400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 msgr2=0x7fe858198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe85000cad0 tx=0x7fe85000ced0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 shutdown_connections 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fe8440a3da0 0x7fe8440a6260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe858198590 0x7fe858198a40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe858107270 0x7fe85819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 --2- 172.21.15.32:0/664437869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8581021a0 0x7fe85819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 >> 172.21.15.32:0/664437869 conn(0x7fe8580fda40 msgr2=0x7fe8580fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 shutdown_connections 2024-04-23T18:26:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.643+0000 7fe85fdd8640 1 -- 172.21.15.32:0/664437869 wait complete. 2024-04-23T18:26:00.651 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:26:00.652 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:26:00.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 -- 172.21.15.32:0/1724176995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103070 msgr2=0x7fd1c4100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 --2- 172.21.15.32:0/1724176995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103070 0x7fd1c4100470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009890 tx=0x7fd1b402a380 comp rx=0 tx=0).stop 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 -- 172.21.15.32:0/1724176995 shutdown_connections 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 --2- 172.21.15.32:0/1724176995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c410ebf0 0x7fd1c4110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 --2- 172.21.15.32:0/1724176995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 0x7fd1c4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 --2- 172.21.15.32:0/1724176995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103070 0x7fd1c4100470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 -- 172.21.15.32:0/1724176995 >> 172.21.15.32:0/1724176995 conn(0x7fd1c40773a0 msgr2=0x7fd1c40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 -- 172.21.15.32:0/1724176995 shutdown_connections 2024-04-23T18:26:00.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.815+0000 7fd1c9472640 1 -- 172.21.15.32:0/1724176995 wait complete. 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.816+0000 7fd1c9472640 1 Processor -- start 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.816+0000 7fd1c9472640 1 -- start start 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 0x7fd1c41a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c4103070 0x7fd1c41a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41145f0 con 0x7fd1c41009b0 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c4114470 con 0x7fd1c410ebf0 2024-04-23T18:26:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c9472640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c4114770 con 0x7fd1c4103070 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c4103070 0x7fd1c41a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56178/0 (socket says 172.21.15.32:56178) 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 -- 172.21.15.32:0/3304981498 learned_addr learned my addr 172.21.15.32:0/3304981498 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 0x7fd1c41a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 -- 172.21.15.32:0/3304981498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c4103070 msgr2=0x7fd1c41a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c4103070 0x7fd1c41a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 -- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 msgr2=0x7fd1c41a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 0x7fd1c41a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 -- 172.21.15.32:0/3304981498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c419b110 con 0x7fd1c410ebf0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.817+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00eb10 tx=0x7fd1bc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.818+0000 7fd1abfff640 1 -- 172.21.15.32:0/3304981498 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00ce20 con 0x7fd1c410ebf0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.818+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c419b400 con 0x7fd1c410ebf0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.818+0000 7fd1abfff640 1 -- 172.21.15.32:0/3304981498 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1bc005020 con 0x7fd1c410ebf0 2024-04-23T18:26:00.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.818+0000 7fd1abfff640 1 -- 172.21.15.32:0/3304981498 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc010690 con 0x7fd1c410ebf0 2024-04-23T18:26:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.818+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c41a76a0 con 0x7fd1c410ebf0 2024-04-23T18:26:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.820+0000 7fd1abfff640 1 -- 172.21.15.32:0/3304981498 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd1bc004b20 con 0x7fd1c410ebf0 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 Processor -- start 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 -- start start 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f47b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1abfff640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 0x7fd1ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560e59ea2370 con 0x560e59f3d9c0 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560e59e3f2e0 con 0x560e59f48090 2024-04-23T18:26:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1cc189740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560e59e9e330 con 0x560e59f3d0a0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:51152/0 (socket says 172.21.15.32:51152) 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c27fc640 1 -- 172.21.15.32:0/3363381617 learned_addr learned my addr 172.21.15.32:0/3363381617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f47b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1abfff640 1 -- 172.21.15.32:0/3304981498 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7fd1bc0d0ed0 con 0x7fd1c410ebf0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 0x7fd1ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.822+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 -- 172.21.15.32:0/3363381617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 msgr2=0x560e59f4a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 -- 172.21.15.32:0/3363381617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 msgr2=0x560e59f47b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f47b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 -- 172.21.15.32:0/3363381617 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e59c80d80 con 0x560e59f3d0a0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009fd0 tx=0x7fd1b8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39ba1732e9ddd60 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 0x7fd1ac0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4035040 tx=0x7fd1b4030340 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1a97fa640 1 -- 172.21.15.32:0/3363381617 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8004050 con 0x560e59f3d0a0 2024-04-23T18:26:00.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1a97fa640 1 -- 172.21.15.32:0/3363381617 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b80054d0 con 0x560e59f3d0a0 2024-04-23T18:26:00.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.823+0000 7fd1a97fa640 1 -- 172.21.15.32:0/3363381617 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8004ba0 con 0x560e59f3d0a0 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- 172.21.15.32:0/3363381617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 msgr2=0x560e59f3d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009fd0 tx=0x7fd1b8005850 comp rx=0 tx=0).stop 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- 172.21.15.32:0/3363381617 shutdown_connections 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f47b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 --2- 172.21.15.32:0/3363381617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f3d480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- 172.21.15.32:0/3363381617 >> 172.21.15.32:0/3363381617 conn(0x560e59f393c0 msgr2=0x560e59f397a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- 172.21.15.32:0/3363381617 shutdown_connections 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- 172.21.15.32:0/3363381617 wait complete. 2024-04-23T18:26:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.824+0000 7fd1cc189740 1 -- start start 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 Processor -- start 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d9c0 0x560e59f45500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560e59f4ae80 con 0x560e59f3d0a0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560e59e3cfb0 con 0x560e59f48090 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560e59f4b000 con 0x560e59f3d9c0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1c2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1c2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44730/0 (socket says 172.21.15.32:44730) 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1cc189740 10 client.0 _ll_register_callbacks cb 0x560e59e3c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1c2ffd640 1 -- 172.21.15.32:0/142761065 learned_addr learned my addr 172.21.15.32:0/142761065 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d9c0 0x560e59f45500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.825+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 -- 172.21.15.32:0/142761065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d9c0 msgr2=0x560e59f45500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d9c0 0x560e59f45500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 -- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 msgr2=0x560e59f45a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 -- 172.21.15.32:0/142761065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e59ce1510 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40095b0 tx=0x7fd1b4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504b14c92158360f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4037040 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7fd1b40035f0 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b80054d0 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fd1b4056050 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1cc189740 10 client.16812 fetch_fsmap learned FSMap version 41 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.826+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x560e59b74670 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b80054d0 con 0x560e59f3d0a0 2024-04-23T18:26:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7fd1b4048960 con 0x560e59f3d0a0 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 10 client.16812 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 10 client.16812 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 20 client.16812 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59f58430 0x560e59f5a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 4 client.16812 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e59eafd90 con 0x560e59f58430 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.827+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59f58430 0x560e59f5a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.828+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59f58430 0x560e59f5a8d0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=923a013cf368f42b server_cookie=75870c2914443fa6 in_seq=0 out_seq=0 2024-04-23T18:26:00.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.828+0000 7fd1a6ff5640 10 client.16812 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.828+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/142761065 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e59eafd90 con 0x560e59f58430 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.828+0000 7fd1a6ff5640 10 client.16812 handle_command_reply: tid=0 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.828+0000 7fd1cc189740 1 client.16812 shutdown 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 msgr2=0x560e59f44fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40095b0 tx=0x7fd1b4005f90 comp rx=0 tx=0).stop 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 shutdown_connections 2024-04-23T18:26:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59f58430 0x560e59f5a8d0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d9c0 0x560e59f45500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1a6ff5640 0 client.16812 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 --2- 172.21.15.32:0/142761065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d0a0 0x560e59f44fc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 >> 172.21.15.32:0/142761065 conn(0x560e59f393c0 msgr2=0x560e59f38600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 shutdown_connections 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 1 -- 172.21.15.32:0/142761065 wait complete. 2024-04-23T18:26:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.829+0000 7fd1cc189740 20 client.16812 trim_cache size 0 max 16384 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 Processor -- start 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 -- start start 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f53d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41ae270 con 0x560e59f3d9c0 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41abb10 con 0x560e59f48090 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1cc189740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41a99c0 con 0x560e59f3d0a0 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44742/0 (socket says 172.21.15.32:44742) 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1c27fc640 1 -- 172.21.15.32:0/1633431393 learned_addr learned my addr 172.21.15.32:0/1633431393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.850+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f53d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 -- 172.21.15.32:0/1633431393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 msgr2=0x560e59f53d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f53d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 -- 172.21.15.32:0/1633431393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 msgr2=0x560e59f45690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 -- 172.21.15.32:0/1633431393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e59f34a80 con 0x560e59f3d9c0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f53d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005c80 tx=0x7fd1b8004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f6aec58ac162ff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1a8ff9640 1 -- 172.21.15.32:0/1633431393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800fa10 con 0x560e59f3d9c0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1a8ff9640 1 -- 172.21.15.32:0/1633431393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b800fbb0 con 0x560e59f3d9c0 2024-04-23T18:26:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.851+0000 7fd1a8ff9640 1 -- 172.21.15.32:0/1633431393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b802b6e0 con 0x560e59f3d9c0 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- 172.21.15.32:0/1633431393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 msgr2=0x560e59f44770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005c80 tx=0x7fd1b8004840 comp rx=0 tx=0).stop 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- 172.21.15.32:0/1633431393 shutdown_connections 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f45690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f44770 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- 172.21.15.32:0/1633431393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f53d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- 172.21.15.32:0/1633431393 >> 172.21.15.32:0/1633431393 conn(0x560e59f393c0 msgr2=0x560e59f44f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- 172.21.15.32:0/1633431393 shutdown_connections 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- 172.21.15.32:0/1633431393 wait complete. 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- start start 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 Processor -- start 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f4bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41aa010 con 0x560e59f3d9c0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41ab440 con 0x560e59f48090 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c41aa6a0 con 0x560e59f3d0a0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.852+0000 7fd1cc189740 10 client.0 _ll_register_callbacks cb 0x7fd1b40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44756/0 (socket says 172.21.15.32:44756) 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f4bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 -- 172.21.15.32:0/627451699 learned_addr learned my addr 172.21.15.32:0/627451699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 -- 172.21.15.32:0/627451699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 msgr2=0x560e59f4bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f4bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 -- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 msgr2=0x560e59f4ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 -- 172.21.15.32:0/627451699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e59fbdd00 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f4bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c37fe640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1c27fc640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009660 tx=0x7fd1b8009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f68112a9508ca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b803a020 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8484+0+0 (secure 0 0 0) 0x7fd1b800f810 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b800fbb0 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fd1b8045050 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1cc189740 10 client.16818 fetch_fsmap learned FSMap version 41 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.853+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x560e59fbefe0 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b800fbb0 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7fd1b802d510 con 0x560e59f3d9c0 2024-04-23T18:26:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 10 client.16818 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 10 client.16818 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 20 client.16818 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59fbf5a0 0x560e59fc19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 4 client.16818 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x560e59ea9c00 con 0x560e59fbf5a0 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59fbf5a0 0x560e59fc19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1a6ff5640 10 client.16818 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:00.854+0000 7fd1c2ffd640 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59fbf5a0 0x560e59fc19c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4429cb1b970514f7 server_cookie=f78cf9c13292067f in_seq=0 out_seq=0 2024-04-23T18:26:01.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:00 smithi032 ceph-mon[29221]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 886 op/s 2024-04-23T18:26:01.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/664437869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1a6ff5640 1 -- 172.21.15.32:0/627451699 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560e59ea9c00 con 0x560e59fbf5a0 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1a6ff5640 10 client.16818 handle_command_reply: tid=0 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 client.16818 shutdown 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 msgr2=0x560e59f4c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009660 tx=0x7fd1b8009a70 comp rx=0 tx=0).stop 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 shutdown_connections 2024-04-23T18:26:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560e59fbf5a0 0x560e59fc19c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e59f48090 0x560e59f4ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560e59f3d9c0 0x560e59f4c4d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 --2- 172.21.15.32:0/627451699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560e59f3d0a0 0x560e59f4bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1a6ff5640 0 client.16818 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.117+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 >> 172.21.15.32:0/627451699 conn(0x560e59f393c0 msgr2=0x560e59f535f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 shutdown_connections 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1cc189740 1 -- 172.21.15.32:0/627451699 wait complete. 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1cc189740 20 client.16818 trim_cache size 0 max 16384 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:26:01.112 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 41, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 38, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "f5377858": { 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 38, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:26:01.113 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:01.114 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 msgr2=0x7fd1ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 0x7fd1ac0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4035040 tx=0x7fd1b4030340 comp rx=0 tx=0).stop 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 msgr2=0x7fd1c419a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00eb10 tx=0x7fd1bc00ef10 comp rx=0 tx=0).stop 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 shutdown_connections 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c410ebf0 0x7fd1c419a990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1c4103070 0x7fd1c41a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd1ac0a3d20 0x7fd1ac0a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 --2- 172.21.15.32:0/3304981498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c41009b0 0x7fd1c41a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 >> 172.21.15.32:0/3304981498 conn(0x7fd1c40773a0 msgr2=0x7fd1c41101a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 shutdown_connections 2024-04-23T18:26:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:01.118+0000 7fd1c9472640 1 -- 172.21.15.32:0/3304981498 wait complete. 2024-04-23T18:26:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:00 smithi040 ceph-mon[32626]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 886 op/s 2024-04-23T18:26:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/664437869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:01.122 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f5377858', quiesce took 1.0 seconds. Will release after: 30.0 2024-04-23T18:26:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:00 smithi064 ceph-mon[33062]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 886 op/s 2024-04-23T18:26:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/664437869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:02 smithi040 ceph-mon[32626]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.4 MiB/s wr, 556 op/s 2024-04-23T18:26:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:02 smithi064 ceph-mon[33062]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.4 MiB/s wr, 556 op/s 2024-04-23T18:26:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:02 smithi032 ceph-mon[29221]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.4 MiB/s wr, 556 op/s 2024-04-23T18:26:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 635.0 s, 1056.6 tps, lat 36.752 ms stddev 42.236 2024-04-23T18:26:05.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:04 smithi040 ceph-mon[32626]: pgmap v340: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.5 MiB/s wr, 574 op/s 2024-04-23T18:26:05.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:04 smithi064 ceph-mon[33062]: pgmap v340: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.5 MiB/s wr, 574 op/s 2024-04-23T18:26:05.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:04 smithi032 ceph-mon[29221]: pgmap v340: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.5 MiB/s wr, 574 op/s 2024-04-23T18:26:07.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:06 smithi040 ceph-mon[32626]: pgmap v341: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 681 op/s 2024-04-23T18:26:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:06 smithi064 ceph-mon[33062]: pgmap v341: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 681 op/s 2024-04-23T18:26:07.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:06 smithi032 ceph-mon[29221]: pgmap v341: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 681 op/s 2024-04-23T18:26:08.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:07 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:08.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:07 smithi040 ceph-mon[32626]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:26:08.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:07 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:08.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:07 smithi064 ceph-mon[33062]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:26:08.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:07 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:08.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:07 smithi032 ceph-mon[29221]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:26:08.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 640.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:09.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:08 smithi040 ceph-mon[32626]: pgmap v343: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.3 MiB/s wr, 494 op/s 2024-04-23T18:26:09.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:08 smithi064 ceph-mon[33062]: pgmap v343: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.3 MiB/s wr, 494 op/s 2024-04-23T18:26:09.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:08 smithi032 ceph-mon[29221]: pgmap v343: 97 pgs: 97 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.3 MiB/s wr, 494 op/s 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.743+0000 7ff55124e640 1 -- 172.21.15.32:0/3044534411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 msgr2=0x7ff54c10ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.743+0000 7ff55124e640 1 --2- 172.21.15.32:0/3044534411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c10ee60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff544009ca0 tx=0x7ff54402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 -- 172.21.15.32:0/3044534411 shutdown_connections 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 --2- 172.21.15.32:0/3044534411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c10ee60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 --2- 172.21.15.32:0/3044534411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c102b70 0x7ff54c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 --2- 172.21.15.32:0/3044534411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c1021c0 0x7ff54c1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 -- 172.21.15.32:0/3044534411 >> 172.21.15.32:0/3044534411 conn(0x7ff54c0fda40 msgr2=0x7ff54c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 -- 172.21.15.32:0/3044534411 shutdown_connections 2024-04-23T18:26:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 -- 172.21.15.32:0/3044534411 wait complete. 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.744+0000 7ff55124e640 1 Processor -- start 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 -- start start 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 0x7ff54c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c19d000 0x7ff54c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c112200 con 0x7ff54c107290 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c112080 con 0x7ff54c1021c0 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff55124e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c112380 con 0x7ff54c19d000 2024-04-23T18:26:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47734/0 (socket says 172.21.15.32:47734) 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 -- 172.21.15.32:0/2675468649 learned_addr learned my addr 172.21.15.32:0/2675468649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 0x7ff54c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c19d000 0x7ff54c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 -- 172.21.15.32:0/2675468649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c19d000 msgr2=0x7ff54c19d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c19d000 0x7ff54c19d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 -- 172.21.15.32:0/2675468649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 msgr2=0x7ff54c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 0x7ff54c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54a59c640 1 -- 172.21.15.32:0/2675468649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c19daf0 con 0x7ff54c107290 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.745+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 0x7ff54c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff54a59c640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d770 tx=0x7ff54000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff533fff640 1 -- 172.21.15.32:0/2675468649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540017070 con 0x7ff54c107290 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54c19dde0 con 0x7ff54c107290 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff533fff640 1 -- 172.21.15.32:0/2675468649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff540009c30 con 0x7ff54c107290 2024-04-23T18:26:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff533fff640 1 -- 172.21.15.32:0/2675468649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54000f620 con 0x7ff54c107290 2024-04-23T18:26:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.746+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54c1a5620 con 0x7ff54c107290 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.748+0000 7ff533fff640 1 -- 172.21.15.32:0/2675468649 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff54001a070 con 0x7ff54c107290 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 Processor -- start 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 -- start start 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff533fff640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 0x7ff5340a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1e80 0x5575e44b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5575e4406970 con 0x5575e44b0c10 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 0x7ff5340a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575e43a34a0 con 0x5575e44a1e80 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff553f67740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5575e4402930 con 0x5575e44a1560 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff533fff640 1 -- 172.21.15.32:0/2675468649 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7ff5400d0210 con 0x7ff54c107290 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:44786/0 (socket says 172.21.15.32:44786) 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47746/0 (socket says 172.21.15.32:47746) 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 -- 172.21.15.32:0/822883431 learned_addr learned my addr 172.21.15.32:0/822883431 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54a59c640 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1e80 0x5575e44b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 -- 172.21.15.32:0/822883431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1e80 msgr2=0x5575e44b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1e80 0x5575e44b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 -- 172.21.15.32:0/822883431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 msgr2=0x5575e44b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.749+0000 7ff54ad9d640 1 -- 172.21.15.32:0/822883431 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e4119210 con 0x5575e44a1560 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.750+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 0x7ff5340a61d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff544006010 tx=0x7ff54402f460 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.750+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0115c0 tx=0x7ff53c011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e94d4181f93296 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.750+0000 7ff5317fa640 1 -- 172.21.15.32:0/822883431 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00f5e0 con 0x5575e44a1560 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.750+0000 7ff5317fa640 1 -- 172.21.15.32:0/822883431 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff53c00fc30 con 0x5575e44a1560 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.750+0000 7ff5317fa640 1 -- 172.21.15.32:0/822883431 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c019bc0 con 0x5575e44a1560 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- 172.21.15.32:0/822883431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 msgr2=0x5575e44a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0115c0 tx=0x7ff53c011a80 comp rx=0 tx=0).stop 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- 172.21.15.32:0/822883431 shutdown_connections 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1e80 0x5575e44b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- 172.21.15.32:0/822883431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1560 0x5575e44a1940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- 172.21.15.32:0/822883431 >> 172.21.15.32:0/822883431 conn(0x5575e449db30 msgr2=0x5575e449df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- 172.21.15.32:0/822883431 shutdown_connections 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- 172.21.15.32:0/822883431 wait complete. 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- start start 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 Processor -- start 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44add80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5575e44b37e0 con 0x5575e44a1e80 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575e43a10b0 con 0x5575e44a1560 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5575e44b3960 con 0x5575e44b0c10 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.751+0000 7ff553f67740 10 client.0 _ll_register_callbacks cb 0x5575e438ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47750/0 (socket says 172.21.15.32:47750) 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54a59c640 1 -- 172.21.15.32:0/3169852218 learned_addr learned my addr 172.21.15.32:0/3169852218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54b59e640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 -- 172.21.15.32:0/3169852218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 msgr2=0x5575e44add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 -- 172.21.15.32:0/3169852218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 msgr2=0x5575e44ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 -- 172.21.15.32:0/3169852218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e41e5620 con 0x5575e44a1560 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44b0550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0115c0 tx=0x7ff53c009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e9ed28e1fa829c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff52eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c027060 con 0x5575e44a1560 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff53c00fc30 con 0x5575e44a1560 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.752+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7ff53c0042d0 con 0x5575e44a1560 2024-04-23T18:26:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff53c0044d0 con 0x5575e44a1560 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 10 client.26360 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5575e44bf770 con 0x5575e44a1560 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff53c00fc30 con 0x5575e44a1560 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7ff53c0050e0 con 0x5575e44a1560 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 10 client.26360 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 10 client.26360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 20 client.26360 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.753+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e44c0c30 0x5575e44c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.754+0000 7ff553f67740 4 client.26360 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.754+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575e4414390 con 0x5575e44c0c30 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.754+0000 7ff54a59c640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e44c0c30 0x5575e44c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff52eff5640 10 client.26360 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff54a59c640 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e44c0c30 0x5575e44c3020 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=440536c6d48ab53f server_cookie=769f62d859a91e01 in_seq=0 out_seq=0 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff52eff5640 1 -- 172.21.15.32:0/3169852218 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575e4414390 con 0x5575e44c0c30 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff52eff5640 10 client.26360 handle_command_reply: tid=0 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff553f67740 1 client.26360 shutdown 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 msgr2=0x5575e44b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44b0550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0115c0 tx=0x7ff53c009d40 comp rx=0 tx=0).stop 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 shutdown_connections 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.755+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e44c0c30 0x5575e44c3020 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 --2- 172.21.15.32:0/3169852218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44b0550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff52eff5640 0 client.26360 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 >> 172.21.15.32:0/3169852218 conn(0x5575e449db30 msgr2=0x5575e44a38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 shutdown_connections 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 1 -- 172.21.15.32:0/3169852218 wait complete. 2024-04-23T18:26:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.756+0000 7ff553f67740 20 client.26360 trim_cache size 0 max 16384 2024-04-23T18:26:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 Processor -- start 2024-04-23T18:26:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 -- start start 2024-04-23T18:26:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44c8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1ae290 con 0x5575e44a1e80 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1abad0 con 0x5575e44a1560 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff553f67740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1a9940 con 0x5575e44b0c10 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47754/0 (socket says 172.21.15.32:47754) 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 -- 172.21.15.32:0/388420006 learned_addr learned my addr 172.21.15.32:0/388420006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54b59e640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 -- 172.21.15.32:0/388420006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 msgr2=0x5575e44c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 -- 172.21.15.32:0/388420006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 msgr2=0x5575e44bc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44bc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54a59c640 1 -- 172.21.15.32:0/388420006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e44ae0a0 con 0x5575e44a1e80 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.771+0000 7ff54b59e640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff54a59c640 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff5400986e0 tx=0x7ff540010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da51521ab888fd3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff530ff9640 1 -- 172.21.15.32:0/388420006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540021040 con 0x5575e44a1e80 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff530ff9640 1 -- 172.21.15.32:0/388420006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5400b5070 con 0x5575e44a1e80 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff530ff9640 1 -- 172.21.15.32:0/388420006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400a0150 con 0x5575e44a1e80 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 -- 172.21.15.32:0/388420006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 msgr2=0x5575e44b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff5400986e0 tx=0x7ff540010c00 comp rx=0 tx=0).stop 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 -- 172.21.15.32:0/388420006 shutdown_connections 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44b0c10 0x5575e44c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44a1e80 0x5575e44b0080 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.772+0000 7ff553f67740 1 --2- 172.21.15.32:0/388420006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- 172.21.15.32:0/388420006 >> 172.21.15.32:0/388420006 conn(0x5575e449db30 msgr2=0x5575e44c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- 172.21.15.32:0/388420006 shutdown_connections 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- 172.21.15.32:0/388420006 wait complete. 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- start start 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 Processor -- start 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1e80 0x5575e44ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1a9f90 con 0x5575e44b0c10 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1ab400 con 0x5575e44a1560 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1aa660 con 0x5575e44a1e80 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff553f67740 10 client.0 _ll_register_callbacks cb 0x7ff53c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff54b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff54b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47756/0 (socket says 172.21.15.32:47756) 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff54b59e640 1 -- 172.21.15.32:0/2201216734 learned_addr learned my addr 172.21.15.32:0/2201216734 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.773+0000 7ff54a59c640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1e80 0x5575e44ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 -- 172.21.15.32:0/2201216734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1e80 msgr2=0x5575e44ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1e80 0x5575e44ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 -- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 msgr2=0x5575e44aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 -- 172.21.15.32:0/2201216734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e44b94a0 con 0x5575e44b0c10 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff54b59e640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff544009ca0 tx=0x7ff54402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2589d04bdf6d8220 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff52eff5640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544004040 con 0x5575e44b0c10 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5400b5070 con 0x5575e44b0c10 2024-04-23T18:26:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7ff5440041e0 con 0x5575e44b0c10 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff5440043e0 con 0x5575e44b0c10 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff553f67740 10 client.16836 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5575e45264d0 con 0x5575e44b0c10 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.774+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5400b5070 con 0x5575e44b0c10 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7ff54405c070 con 0x5575e44b0c10 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 10 client.16836 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 20 client.16836 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e4526ce0 0x5575e4529100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 4 client.16836 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575e440e200 con 0x5575e4526ce0 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e4526ce0 0x5575e4529100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff52eff5640 10 client.16836 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.775+0000 7ff54ad9d640 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e4526ce0 0x5575e4529100 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c570a789384ee39d server_cookie=c7b4b1d0313c3376 in_seq=0 out_seq=0 2024-04-23T18:26:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff52eff5640 1 -- 172.21.15.32:0/2201216734 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575e440e200 con 0x5575e4526ce0 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff52eff5640 10 client.16836 handle_command_reply: tid=0 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 client.16836 shutdown 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 msgr2=0x5575e44a9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff544009ca0 tx=0x7ff54402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 shutdown_connections 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5575e4526ce0 0x5575e4529100 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5575e44b0c10 0x5575e44a9ab0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575e44a1e80 0x5575e44ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 --2- 172.21.15.32:0/2201216734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575e44a1560 0x5575e44aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff52eff5640 0 client.16836 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 >> 172.21.15.32:0/2201216734 conn(0x5575e449db30 msgr2=0x5575e44a8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 shutdown_connections 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 1 -- 172.21.15.32:0/2201216734 wait complete. 2024-04-23T18:26:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.798+0000 7ff553f67740 20 client.16836 trim_cache size 0 max 16384 2024-04-23T18:26:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:26:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:26:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:26:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 msgr2=0x7ff5340a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 0x7ff5340a61d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff544006010 tx=0x7ff54402f460 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 msgr2=0x7ff54c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d770 tx=0x7ff54000dc30 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 shutdown_connections 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff54c19d000 0x7ff54c19d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff5340a3d10 0x7ff5340a61d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c107290 0x7ff54c198c80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 --2- 172.21.15.32:0/2675468649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff54c1021c0 0x7ff54c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 >> 172.21.15.32:0/2675468649 conn(0x7ff54c0fda40 msgr2=0x7ff54c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 shutdown_connections 2024-04-23T18:26:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:10.799+0000 7ff55124e640 1 -- 172.21.15.32:0/2675468649 wait complete. 2024-04-23T18:26:11.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:10 smithi040 ceph-mon[32626]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-23T18:26:11.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:10 smithi040 ceph-mon[32626]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:26:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:10 smithi064 ceph-mon[33062]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-23T18:26:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:10 smithi064 ceph-mon[33062]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:26:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:10 smithi032 ceph-mon[29221]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-23T18:26:11.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:10 smithi032 ceph-mon[29221]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:26:13.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:12 smithi040 ceph-mon[32626]: pgmap v346: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-23T18:26:13.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:12 smithi064 ceph-mon[33062]: pgmap v346: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-23T18:26:13.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:12 smithi032 ceph-mon[29221]: pgmap v346: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-23T18:26:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 645.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:15.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:14 smithi040 ceph-mon[32626]: pgmap v347: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-23T18:26:15.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:14 smithi064 ceph-mon[33062]: pgmap v347: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-23T18:26:15.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:14 smithi032 ceph-mon[29221]: pgmap v347: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-23T18:26:17.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:16 smithi040 ceph-mon[32626]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 37 MiB/s wr, 1.45k op/s 2024-04-23T18:26:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:16 smithi064 ceph-mon[33062]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 37 MiB/s wr, 1.45k op/s 2024-04-23T18:26:17.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:16 smithi032 ceph-mon[29221]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 37 MiB/s wr, 1.45k op/s 2024-04-23T18:26:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 650.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:18 smithi040 ceph-mon[32626]: pgmap v349: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:18 smithi064 ceph-mon[33062]: pgmap v349: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:19.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:18 smithi032 ceph-mon[29221]: pgmap v349: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:21.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:20 smithi040 ceph-mon[32626]: pgmap v350: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:20 smithi064 ceph-mon[33062]: pgmap v350: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:20 smithi032 ceph-mon[29221]: pgmap v350: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 32 MiB/s wr, 1.27k op/s 2024-04-23T18:26:23.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:22 smithi040 ceph-mon[32626]: pgmap v351: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.17k op/s 2024-04-23T18:26:23.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:22 smithi064 ceph-mon[33062]: pgmap v351: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.17k op/s 2024-04-23T18:26:23.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:22 smithi032 ceph-mon[29221]: pgmap v351: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.17k op/s 2024-04-23T18:26:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 655.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 -- 172.21.15.32:0/993465886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c1031e0 msgr2=0x7fbc2c1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 --2- 172.21.15.32:0/993465886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c1035c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009900 tx=0x7fbc1c02a480 comp rx=0 tx=0).stop 2024-04-23T18:26:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 -- 172.21.15.32:0/993465886 shutdown_connections 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 --2- 172.21.15.32:0/993465886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 0x7fbc2c10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 --2- 172.21.15.32:0/993465886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c103b00 0x7fbc2c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 --2- 172.21.15.32:0/993465886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c1035c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 -- 172.21.15.32:0/993465886 >> 172.21.15.32:0/993465886 conn(0x7fbc2c077f70 msgr2=0x7fbc2c078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 -- 172.21.15.32:0/993465886 shutdown_connections 2024-04-23T18:26:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.900+0000 7fbc31f01640 1 -- 172.21.15.32:0/993465886 wait complete. 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 Processor -- start 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 -- start start 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 0x7fbc2c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c112200 con 0x7fbc2c103b00 2024-04-23T18:26:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c112080 con 0x7fbc2c1031e0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc31f01640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c112380 con 0x7fbc2c10c5b0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.901+0000 7fbc2affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40796/0 (socket says 172.21.15.32:40796) 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 -- 172.21.15.32:0/745546660 learned_addr learned my addr 172.21.15.32:0/745546660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 0x7fbc2c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 -- 172.21.15.32:0/745546660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 msgr2=0x7fbc2c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 0x7fbc2c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 -- 172.21.15.32:0/745546660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c1031e0 msgr2=0x7fbc2c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 -- 172.21.15.32:0/745546660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c19d750 con 0x7fbc2c103b00 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc2affd640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c890 tx=0x7fbc2000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc28ff9640 1 -- 172.21.15.32:0/745546660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20004220 con 0x7fbc2c103b00 2024-04-23T18:26:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc28ff9640 1 -- 172.21.15.32:0/745546660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc200043c0 con 0x7fbc2c103b00 2024-04-23T18:26:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc28ff9640 1 -- 172.21.15.32:0/745546660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20004a00 con 0x7fbc2c103b00 2024-04-23T18:26:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c19da40 con 0x7fbc2c103b00 2024-04-23T18:26:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.902+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c1a52d0 con 0x7fbc2c103b00 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc28ff9640 1 -- 172.21.15.32:0/745546660 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fbc200079b0 con 0x7fbc2c103b00 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc34c18740 1 Processor -- start 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc34c18740 1 -- start start 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc28ff9640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 0x7fbc140a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.905+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e86d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc34c18740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56457e7c0970 con 0x56457e85be40 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc34c18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56457e75d4a0 con 0x56457e86ac10 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc34c18740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56457e7bc930 con 0x56457e85b520 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 0x7fbc140a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:57650/0 (socket says 172.21.15.32:57650) 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/2624012031 learned_addr learned my addr 172.21.15.32:0/2624012031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc28ff9640 1 -- 172.21.15.32:0/745546660 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fbc200d0480 con 0x7fbc2c103b00 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e86d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/2624012031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 msgr2=0x56457e86d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e86d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/2624012031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 msgr2=0x56457e86a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/2624012031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457e4d3210 con 0x56457e85b520 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2affd640 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 0x7fbc140a61d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06b6b0 tx=0x7fbc24009e60 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.906+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009900 tx=0x7fbc1c032e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8603ae7b1f5bb590 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc11ffb640 1 -- 172.21.15.32:0/2624012031 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c040070 con 0x56457e85b520 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc11ffb640 1 -- 172.21.15.32:0/2624012031 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1c0316a0 con 0x56457e85b520 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc11ffb640 1 -- 172.21.15.32:0/2624012031 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c035040 con 0x56457e85b520 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 -- 172.21.15.32:0/2624012031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 msgr2=0x56457e85b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009900 tx=0x7fbc1c032e60 comp rx=0 tx=0).stop 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 -- 172.21.15.32:0/2624012031 shutdown_connections 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e86d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2624012031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85b520 0x56457e85b900 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 -- 172.21.15.32:0/2624012031 >> 172.21.15.32:0/2624012031 conn(0x56457e857b30 msgr2=0x56457e857f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 -- 172.21.15.32:0/2624012031 shutdown_connections 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.907+0000 7fbc34c18740 1 -- 172.21.15.32:0/2624012031 wait complete. 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 -- start start 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 Processor -- start 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 0x56457e867720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e867c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56457e86d7e0 con 0x56457e85b520 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56457e75b0b0 con 0x56457e86ac10 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56457e86d960 con 0x56457e85be40 2024-04-23T18:26:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc34c18740 10 client.0 _ll_register_callbacks cb 0x56457e748d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40816/0 (socket says 172.21.15.32:40816) 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/782742695 learned_addr learned my addr 172.21.15.32:0/782742695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.908+0000 7fbc2affd640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 0x56457e867720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e867c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/782742695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 msgr2=0x56457e867720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 0x56457e867720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 msgr2=0x56457e867c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e867c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/782742695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56457e59f620 con 0x56457e85b520 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2affd640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 0x56457e867720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0096c0 tx=0x7fbc1c02ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec5a4cd0878b71 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc0affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c035070 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fbc1c004230 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc1c0316a0 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbc1c055050 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc34c18740 10 client.16848 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.909+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56457e879680 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1c0316a0 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7fbc1c03c440 con 0x56457e85b520 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 10 client.16848 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 10 client.16848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 20 client.16848 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e87a9d0 0x56457e87cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 4 client.16848 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56457e7ce390 con 0x56457e87a9d0 2024-04-23T18:26:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc2affd640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e87a9d0 0x56457e87cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc2affd640 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e87a9d0 0x56457e87cdc0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c983404069d4f356 server_cookie=5cef0f1131d4046 in_seq=0 out_seq=0 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.910+0000 7fbc0affd640 10 client.16848 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc0affd640 1 -- 172.21.15.32:0/782742695 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56457e7ce390 con 0x56457e87a9d0 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc0affd640 10 client.16848 handle_command_reply: tid=0 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 client.16848 shutdown 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 msgr2=0x56457e86a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0096c0 tx=0x7fbc1c02ab80 comp rx=0 tx=0).stop 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 shutdown_connections 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e87a9d0 0x56457e87cdc0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e86ac10 0x56457e867c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e85be40 0x56457e867720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 --2- 172.21.15.32:0/782742695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e86a430 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 >> 172.21.15.32:0/782742695 conn(0x56457e857b30 msgr2=0x56457e85d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc0affd640 0 client.16848 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.911+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 shutdown_connections 2024-04-23T18:26:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.912+0000 7fbc34c18740 1 -- 172.21.15.32:0/782742695 wait complete. 2024-04-23T18:26:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.912+0000 7fbc34c18740 20 client.16848 trim_cache size 0 max 16384 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.926+0000 7fbc34c18740 1 Processor -- start 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 -- start start 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 0x56457e862f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e866700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1ae2a0 con 0x56457e85b520 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1abae0 con 0x56457e85be40 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc34c18740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1a9950 con 0x56457e86ac10 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40822/0 (socket says 172.21.15.32:40822) 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/3640479574 learned_addr learned my addr 172.21.15.32:0/3640479574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e866700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2affd640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 0x56457e862f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/3640479574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 msgr2=0x56457e866700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e866700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/3640479574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 msgr2=0x56457e862f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 0x56457e862f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 -- 172.21.15.32:0/3640479574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457e85bc70 con 0x56457e85b520 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e866700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2affd640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 0x56457e862f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c03c810 tx=0x7fbc1c032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118659f250dd6b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc117fa640 1 -- 172.21.15.32:0/3640479574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c035040 con 0x56457e85b520 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc117fa640 1 -- 172.21.15.32:0/3640479574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1c05e070 con 0x56457e85b520 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.927+0000 7fbc117fa640 1 -- 172.21.15.32:0/3640479574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c040040 con 0x56457e85b520 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- 172.21.15.32:0/3640479574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 msgr2=0x56457e876710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c03c810 tx=0x7fbc1c032d40 comp rx=0 tx=0).stop 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- 172.21.15.32:0/3640479574 shutdown_connections 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e866700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85be40 0x56457e862f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 --2- 172.21.15.32:0/3640479574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85b520 0x56457e876710 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- 172.21.15.32:0/3640479574 >> 172.21.15.32:0/3640479574 conn(0x56457e857b30 msgr2=0x56457e865ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- 172.21.15.32:0/3640479574 shutdown_connections 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- 172.21.15.32:0/3640479574 wait complete. 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.928+0000 7fbc34c18740 1 -- start start 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 Processor -- start 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85b520 0x56457e872e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e86e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1a9fa0 con 0x56457e85be40 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1ab410 con 0x56457e85b520 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c1aa670 con 0x56457e86ac10 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc34c18740 10 client.0 _ll_register_callbacks cb 0x7fbc1c032db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40834/0 (socket says 172.21.15.32:40834) 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 -- 172.21.15.32:0/2648433110 learned_addr learned my addr 172.21.15.32:0/2648433110 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85b520 0x56457e872e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e86e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 -- 172.21.15.32:0/2648433110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 msgr2=0x56457e86e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e86e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 -- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85b520 msgr2=0x56457e872e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85b520 0x56457e872e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2affd640 1 -- 172.21.15.32:0/2648433110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56457e86ed20 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.929+0000 7fbc2bfff640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e86e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc2affd640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbc20098ec0 tx=0x7fbc2009bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b05e5ed145ec597 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2006be60 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fbc200cde70 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc1c05e070 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbc2006e3d0 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 10 client.16854 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56457e8e0a60 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1c05e070 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7fbc200b5bc0 con 0x56457e85be40 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 10 client.16854 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 10 client.16854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 20 client.16854 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e8e1270 0x56457e8e3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 4 client.16854 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.930+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56457e7c8200 con 0x56457e8e1270 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.931+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e8e1270 0x56457e8e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.931+0000 7fbc2b7fe640 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e8e1270 0x56457e8e3690 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6a441d8dc60f server_cookie=8752891ada068782 in_seq=0 out_seq=0 2024-04-23T18:26:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.931+0000 7fbc0affd640 10 client.16854 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc0affd640 1 -- 172.21.15.32:0/2648433110 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56457e7c8200 con 0x56457e8e1270 2024-04-23T18:26:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc0affd640 10 client.16854 handle_command_reply: tid=0 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 client.16854 shutdown 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 msgr2=0x56457e86e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbc20098ec0 tx=0x7fbc2009bec0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 shutdown_connections 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56457e8e1270 0x56457e8e3690 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56457e86ac10 0x56457e86e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56457e85be40 0x56457e86e1a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc0affd640 0 client.16854 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 --2- 172.21.15.32:0/2648433110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56457e85b520 0x56457e872e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 >> 172.21.15.32:0/2648433110 conn(0x56457e857b30 msgr2=0x56457e866880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 shutdown_connections 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 1 -- 172.21.15.32:0/2648433110 wait complete. 2024-04-23T18:26:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.950+0000 7fbc34c18740 20 client.16854 trim_cache size 0 max 16384 2024-04-23T18:26:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:26:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:26:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:26:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 msgr2=0x7fbc140a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 0x7fbc140a61d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06b6b0 tx=0x7fbc24009e60 comp rx=0 tx=0).stop 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 msgr2=0x7fbc2c198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c890 tx=0x7fbc2000cd50 comp rx=0 tx=0).stop 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 shutdown_connections 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc2c10c5b0 0x7fbc2c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fbc140a3d10 0x7fbc140a61d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2c103b00 0x7fbc2c198ca0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 --2- 172.21.15.32:0/745546660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc2c1031e0 0x7fbc2c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 >> 172.21.15.32:0/745546660 conn(0x7fbc2c077f70 msgr2=0x7fbc2c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 shutdown_connections 2024-04-23T18:26:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:23.951+0000 7fbc31f01640 1 -- 172.21.15.32:0/745546660 wait complete. 2024-04-23T18:26:25.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:24 smithi040 ceph-mon[32626]: pgmap v352: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 1.06k op/s 2024-04-23T18:26:25.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:24 smithi064 ceph-mon[33062]: pgmap v352: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 1.06k op/s 2024-04-23T18:26:25.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:24 smithi032 ceph-mon[29221]: pgmap v352: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 1.06k op/s 2024-04-23T18:26:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:26 smithi040 ceph-mon[32626]: pgmap v353: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 MiB/s wr, 1.03k op/s 2024-04-23T18:26:27.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:26 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:26 smithi064 ceph-mon[33062]: pgmap v353: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 MiB/s wr, 1.03k op/s 2024-04-23T18:26:27.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:26 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:26 smithi032 ceph-mon[29221]: pgmap v353: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 MiB/s wr, 1.03k op/s 2024-04-23T18:26:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:26 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:28.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: pgmap v354: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:26:28.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:28.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:28.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: pgmap v354: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:28.616 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: pgmap v354: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:26:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:28.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:28.672 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:26:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 660.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:30.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:30 smithi064 ceph-mon[33062]: pgmap v355: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-23T18:26:31.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:30 smithi032 ceph-mon[29221]: pgmap v355: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-23T18:26:31.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:30 smithi040 ceph-mon[32626]: pgmap v355: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-23T18:26:31.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:31.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 -- 172.21.15.32:0/2727669466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c1008a0 msgr2=0x7efe5c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 --2- 172.21.15.32:0/2727669466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c1008a0 0x7efe5c100d20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efe4c009900 tx=0x7efe4c02a480 comp rx=0 tx=0).stop 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 -- 172.21.15.32:0/2727669466 shutdown_connections 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 --2- 172.21.15.32:0/2727669466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 --2- 172.21.15.32:0/2727669466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c1008a0 0x7efe5c100d20 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 --2- 172.21.15.32:0/2727669466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c106710 0x7efe5c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 -- 172.21.15.32:0/2727669466 >> 172.21.15.32:0/2727669466 conn(0x7efe5c076130 msgr2=0x7efe5c076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 -- 172.21.15.32:0/2727669466 shutdown_connections 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.246+0000 7efe63479640 1 -- 172.21.15.32:0/2727669466 wait complete. 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 Processor -- start 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 -- start start 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1008a0 0x7efe5c197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1110a0 con 0x7efe5c106710 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c110f20 con 0x7efe5c101260 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe63479640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c111220 con 0x7efe5c1008a0 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe61a16640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe61a16640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37944/0 (socket says 172.21.15.32:37944) 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.247+0000 7efe61a16640 1 -- 172.21.15.32:0/2248004073 learned_addr learned my addr 172.21.15.32:0/2248004073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61215640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1008a0 0x7efe5c197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe60a14640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 -- 172.21.15.32:0/2248004073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1008a0 msgr2=0x7efe5c197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1008a0 0x7efe5c197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 -- 172.21.15.32:0/2248004073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 msgr2=0x7efe5c197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 -- 172.21.15.32:0/2248004073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c1a6b80 con 0x7efe5c106710 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe60a14640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe61a16640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efe5400eaa0 tx=0x7efe5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400cdb0 con 0x7efe5c106710 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe54005020 con 0x7efe5c106710 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5c1a6e70 con 0x7efe5c106710 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540116d0 con 0x7efe5c106710 2024-04-23T18:26:31.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.248+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5c1a7340 con 0x7efe5c106710 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.249+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe5c069250 con 0x7efe5c106710 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.250+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7efe540051d0 con 0x7efe5c106710 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.251+0000 7efe527fc640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 0x7efe480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.251+0000 7efe61215640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 0x7efe480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.251+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7efe540d09c0 con 0x7efe5c106710 2024-04-23T18:26:31.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.252+0000 7efe61215640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 0x7efe480a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe58005fd0 tx=0x7efe5800b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.256+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe54016030 con 0x7efe5c106710 2024-04-23T18:26:31.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.484+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe5c02d5c0 con 0x7efe5c106710 2024-04-23T18:26:31.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.485+0000 7efe527fc640 1 -- 172.21.15.32:0/2248004073 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11855 (secure 0 0 0) 0x7efe5409d020 con 0x7efe5c106710 2024-04-23T18:26:31.478 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:26:31.478 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:26:05.875467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:26:31.478 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-04-23T18:26:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 msgr2=0x7efe480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 0x7efe480a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe58005fd0 tx=0x7efe5800b390 comp rx=0 tx=0).stop 2024-04-23T18:26:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 msgr2=0x7efe5c197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efe5400eaa0 tx=0x7efe5400ef60 comp rx=0 tx=0).stop 2024-04-23T18:26:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 shutdown_connections 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7efe480a3da0 0x7efe480a6260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe5c106710 0x7efe5c197d90 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe5c101260 0x7efe5c197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 --2- 172.21.15.32:0/2248004073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1008a0 0x7efe5c197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 >> 172.21.15.32:0/2248004073 conn(0x7efe5c076130 msgr2=0x7efe5c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 shutdown_connections 2024-04-23T18:26:31.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.487+0000 7efe63479640 1 -- 172.21.15.32:0/2248004073 wait complete. 2024-04-23T18:26:31.492 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id f5377858 --release --await' 2024-04-23T18:26:31.492 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id f5377858 --release --await 2024-04-23T18:26:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 -- 172.21.15.32:0/2492886895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258103400 msgr2=0x7f225810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 --2- 172.21.15.32:0/2492886895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258103400 0x7f225810de00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2250009cd0 tx=0x7f225002a8e0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 -- 172.21.15.32:0/2492886895 shutdown_connections 2024-04-23T18:26:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 --2- 172.21.15.32:0/2492886895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258103400 0x7f225810de00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 --2- 172.21.15.32:0/2492886895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 0x7f2258102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 --2- 172.21.15.32:0/2492886895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258101180 0x7f2258101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 -- 172.21.15.32:0/2492886895 >> 172.21.15.32:0/2492886895 conn(0x7f2258077f70 msgr2=0x7f2258078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 -- 172.21.15.32:0/2492886895 shutdown_connections 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.608+0000 7f225d9cd640 1 -- 172.21.15.32:0/2492886895 wait complete. 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 Processor -- start 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 -- start start 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 0x7f22580fe330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 0x7f22580fe870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2258111250 con 0x7f2258101180 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22581110d0 con 0x7f2258103400 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f225d9cd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22581113d0 con 0x7f2258102a60 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37966/0 (socket says 172.21.15.32:37966) 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 -- 172.21.15.32:0/484117988 learned_addr learned my addr 172.21.15.32:0/484117988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f224ffff640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 0x7f22580fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f22577fe640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 0x7f22580fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 -- 172.21.15.32:0/484117988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 msgr2=0x7f22580fe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 0x7f22580fe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 -- 172.21.15.32:0/484117988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 msgr2=0x7f22580fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 0x7f22580fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f2256ffd640 1 -- 172.21.15.32:0/484117988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22581a6d00 con 0x7f2258101180 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.609+0000 7f22577fe640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 0x7f22580fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f2256ffd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f224800c890 tx=0x7f224800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f225c9cb640 1 -- 172.21.15.32:0/484117988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248004220 con 0x7f2258101180 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f225c9cb640 1 -- 172.21.15.32:0/484117988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22480043c0 con 0x7f2258101180 2024-04-23T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22581a6ff0 con 0x7f2258101180 2024-04-23T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f225c9cb640 1 -- 172.21.15.32:0/484117988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248004a00 con 0x7f2258101180 2024-04-23T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.610+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22581a74b0 con 0x7f2258101180 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.612+0000 7f225c9cb640 1 -- 172.21.15.32:0/484117988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f22480079b0 con 0x7f2258101180 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.612+0000 7f22607c1740 1 Processor -- start 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.612+0000 7f22607c1740 1 -- start start 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9bcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f225c9cb640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562b1c9176d0 con 0x562b1c9b2490 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b1c8b41a0 con 0x562b1c9b2db0 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22607c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562b1c913690 con 0x562b1c9bd480 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37972/0 (socket says 172.21.15.32:37972) 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 -- 172.21.15.32:0/2655930859 learned_addr learned my addr 172.21.15.32:0/2655930859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f224ffff640 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f225c9cb640 1 -- 172.21.15.32:0/484117988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f22480d05e0 con 0x7f2258101180 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f2256ffd640 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9bcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 -- 172.21.15.32:0/2655930859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 msgr2=0x562b1c9bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 -- 172.21.15.32:0/2655930859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 msgr2=0x562b1c9bcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9bcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 -- 172.21.15.32:0/2655930859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b1c8be5b0 con 0x562b1c9b2490 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.613+0000 7f22577fe640 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2250009430 tx=0x7f2250031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c2a628b94dcc89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22577fe640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 0x7f22440a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f225002ae00 tx=0x7f2250048000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f224dffb640 1 -- 172.21.15.32:0/2655930859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250004610 con 0x562b1c9b2490 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f224dffb640 1 -- 172.21.15.32:0/2655930859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2250036040 con 0x562b1c9b2490 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f224dffb640 1 -- 172.21.15.32:0/2655930859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225004a450 con 0x562b1c9b2490 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 -- 172.21.15.32:0/2655930859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 msgr2=0x562b1c9b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2250009430 tx=0x7f2250031af0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 -- 172.21.15.32:0/2655930859 shutdown_connections 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9bcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 --2- 172.21.15.32:0/2655930859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9b2870 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 -- 172.21.15.32:0/2655930859 >> 172.21.15.32:0/2655930859 conn(0x562b1c9ae7b0 msgr2=0x562b1c9aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 -- 172.21.15.32:0/2655930859 shutdown_connections 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.614+0000 7f22607c1740 1 -- 172.21.15.32:0/2655930859 wait complete. 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 -- start start 2024-04-23T18:26:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 Processor -- start 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 0x562b1c9ba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 0x562b1c9ba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562b1c9c02b0 con 0x562b1c9b2490 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b1c8b1b20 con 0x562b1c9bd480 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562b1c9c0430 con 0x562b1c9b2db0 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22607c1740 10 client.0 _ll_register_callbacks cb 0x562b1c8b1530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37986/0 (socket says 172.21.15.32:37986) 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f2256ffd640 1 -- 172.21.15.32:0/1951757322 learned_addr learned my addr 172.21.15.32:0/1951757322 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f22577fe640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 0x562b1c9ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.615+0000 7f224ffff640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 0x562b1c9ba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 -- 172.21.15.32:0/1951757322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 msgr2=0x562b1c9ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 0x562b1c9ba1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 -- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 msgr2=0x562b1c9ba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 0x562b1c9ba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 -- 172.21.15.32:0/1951757322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b1c629920 con 0x562b1c9b2490 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22577fe640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 0x562b1c9ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f224ffff640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 0x562b1c9ba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2256ffd640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f22480cddb0 tx=0x7f224809bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcec450603a0d4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224801d040 con 0x562b1c9b2490 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f224809b0d0 con 0x562b1c9b2490 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2250036040 con 0x562b1c9b2490 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f22480b7e60 con 0x562b1c9b2490 2024-04-23T18:26:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 10 client.16872 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562b1c6f5d30 con 0x562b1c9b2490 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2250036040 con 0x562b1c9b2490 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f224801ed00 con 0x562b1c9b2490 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 10 client.16872 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 10 client.16872 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 20 client.16872 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1c9cd510 0x562b1c9cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 4 client.16872 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.616+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b1c8d5500 con 0x562b1c9cd510 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.617+0000 7f224ffff640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1c9cd510 0x562b1c9cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.617+0000 7f224ffff640 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1c9cd510 0x562b1c9cf9b0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e22fb3afcae2d65 server_cookie=d3c877facd5b8948 in_seq=0 out_seq=0 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.617+0000 7f2236ffd640 10 client.16872 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f2236ffd640 1 -- 172.21.15.32:0/1951757322 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b1c8d5500 con 0x562b1c9cd510 2024-04-23T18:26:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f2236ffd640 10 client.16872 handle_command_reply: tid=0 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 client.16872 shutdown 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 msgr2=0x562b1c9bcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f22480cddb0 tx=0x7f224809bfc0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 shutdown_connections 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1c9cd510 0x562b1c9cf9b0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9bd480 0x562b1c9ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9b2db0 0x562b1c9ba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f2236ffd640 0 client.16872 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 --2- 172.21.15.32:0/1951757322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9bcf00 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 >> 172.21.15.32:0/1951757322 conn(0x562b1c9ae7b0 msgr2=0x562b1c9ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 shutdown_connections 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.618+0000 7f22607c1740 1 -- 172.21.15.32:0/1951757322 wait complete. 2024-04-23T18:26:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.619+0000 7f22607c1740 20 client.16872 trim_cache size 0 max 16384 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 Processor -- start 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 -- start start 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9b9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22581b2590 con 0x562b1c9b2490 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22581afdd0 con 0x562b1c9b2db0 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f22607c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22581adc40 con 0x562b1c9bd480 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37990/0 (socket says 172.21.15.32:37990) 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 -- 172.21.15.32:0/3393608274 learned_addr learned my addr 172.21.15.32:0/3393608274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f224ffff640 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9b9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 -- 172.21.15.32:0/3393608274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 msgr2=0x562b1c9bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 -- 172.21.15.32:0/3393608274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 msgr2=0x562b1c9b9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9b9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 -- 172.21.15.32:0/3393608274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b1c9b8740 con 0x562b1c9b2490 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f224ffff640 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9b9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f2256ffd640 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2248004770 tx=0x7f224809c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afa40e3fb8d23e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.636+0000 7f224d7fa640 1 -- 172.21.15.32:0/3393608274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224801d040 con 0x562b1c9b2490 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f224d7fa640 1 -- 172.21.15.32:0/3393608274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22480a5210 con 0x562b1c9b2490 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f224d7fa640 1 -- 172.21.15.32:0/3393608274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224809b030 con 0x562b1c9b2490 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- 172.21.15.32:0/3393608274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 msgr2=0x562b1c9c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2248004770 tx=0x7f224809c030 comp rx=0 tx=0).stop 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- 172.21.15.32:0/3393608274 shutdown_connections 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9b9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 --2- 172.21.15.32:0/3393608274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c9170 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- 172.21.15.32:0/3393608274 >> 172.21.15.32:0/3393608274 conn(0x562b1c9ae7b0 msgr2=0x562b1c9b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- 172.21.15.32:0/3393608274 shutdown_connections 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- 172.21.15.32:0/3393608274 wait complete. 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.637+0000 7f22607c1740 1 -- start start 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 Processor -- start 2024-04-23T18:26:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9c0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22581ae290 con 0x562b1c9b2490 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22581af700 con 0x562b1c9b2db0 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22581ae960 con 0x562b1c9bd480 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22607c1740 10 client.0 _ll_register_callbacks cb 0x7f224809c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37996/0 (socket says 172.21.15.32:37996) 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 -- 172.21.15.32:0/2030275647 learned_addr learned my addr 172.21.15.32:0/2030275647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f224ffff640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 -- 172.21.15.32:0/2030275647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 msgr2=0x562b1c9c0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f22577fe640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9c0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9c0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 -- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 msgr2=0x562b1c9c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f2256ffd640 1 -- 172.21.15.32:0/2030275647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b1c9c1040 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.638+0000 7f224ffff640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2256ffd640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f22480ce350 tx=0x7f2248007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419a75d880f523f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224801d070 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f224809bc00 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22480a5210 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2248082050 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 10 client.16878 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562b1ca34750 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22480a5210 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f22480982f0 con 0x562b1c9b2490 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 10 client.16878 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 10 client.16878 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 20 client.16878 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1ca34c20 0x562b1ca37010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 4 client.16878 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f5377858", "release": true, "await": true}] 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.639+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f5377858", "release": true, "await": true}) -- 0x562b1c9250f0 con 0x562b1ca34c20 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.640+0000 7f224ffff640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1ca34c20 0x562b1ca37010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.640+0000 7f224ffff640 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1ca34c20 0x562b1ca37010 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9a1027a4d0f9d5 server_cookie=8269b21a48b31ad4 in_seq=0 out_seq=0 2024-04-23T18:26:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.640+0000 7f2236ffd640 10 client.16878 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f2236ffd640 1 -- 172.21.15.32:0/2030275647 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x562b1c9250f0 con 0x562b1ca34c20 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f2236ffd640 10 client.16878 handle_command_reply: tid=0 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 client.16878 shutdown 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 msgr2=0x562b1c9c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f22480ce350 tx=0x7f2248007eb0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 shutdown_connections 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562b1ca34c20 0x562b1ca37010 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b1c9bd480 0x562b1c9c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b1c9b2db0 0x562b1c9c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f2236ffd640 0 client.16878 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 --2- 172.21.15.32:0/2030275647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562b1c9b2490 0x562b1c9c5690 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 >> 172.21.15.32:0/2030275647 conn(0x562b1c9ae7b0 msgr2=0x562b1c9c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 shutdown_connections 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.649+0000 7f22607c1740 1 -- 172.21.15.32:0/2030275647 wait complete. 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f22607c1740 20 client.16878 trim_cache size 0 max 16384 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 42, 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 40, 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "f5377858": { 2024-04-23T18:26:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 40, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:31.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:31.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:31.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:31.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 msgr2=0x7f22440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 0x7f22440a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f225002ae00 tx=0x7f2250048000 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 msgr2=0x7f22580fddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f224800c890 tx=0x7f224800cd50 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 shutdown_connections 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2258103400 0x7f22580fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2258102a60 0x7f22580fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 --2- 172.21.15.32:0/484117988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258101180 0x7f22580fddf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 >> 172.21.15.32:0/484117988 conn(0x7f2258077f70 msgr2=0x7f225806a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 shutdown_connections 2024-04-23T18:26:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.650+0000 7f225d9cd640 1 -- 172.21.15.32:0/484117988 wait complete. 2024-04-23T18:26:31.656 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f5377858', seconds elapsed: 31.6 2024-04-23T18:26:31.656 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:26:31.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:31.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:31 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2248004073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:31.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 -- 172.21.15.32:0/134940631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 msgr2=0x7f5d48108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/134940631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d48108f50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009900 tx=0x7f5d3802a480 comp rx=0 tx=0).stop 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 -- 172.21.15.32:0/134940631 shutdown_connections 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/134940631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48116fe0 0x7f5d481193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/134940631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48109490 0x7f5d481098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/134940631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d48108f50 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 -- 172.21.15.32:0/134940631 >> 172.21.15.32:0/134940631 conn(0x7f5d4806e540 msgr2=0x7f5d4806e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.812+0000 7f5d4cf22640 1 -- 172.21.15.32:0/134940631 shutdown_connections 2024-04-23T18:26:31.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.813+0000 7f5d4cf22640 1 -- 172.21.15.32:0/134940631 wait complete. 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.813+0000 7f5d4cf22640 1 Processor -- start 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.813+0000 7f5d4cf22640 1 -- start start 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 0x7f5d481a3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d481a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4811cbc0 con 0x7f5d4810c6e0 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4811ca40 con 0x7f5d48109490 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4cf22640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4811cd40 con 0x7f5d48116fe0 2024-04-23T18:26:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d45d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d481a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 0x7f5d481a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39610/0 (socket says 172.21.15.32:39610) 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d45d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d481a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38010/0 (socket says 172.21.15.32:38010) 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 -- 172.21.15.32:0/3564302501 learned_addr learned my addr 172.21.15.32:0/3564302501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 -- 172.21.15.32:0/3564302501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 msgr2=0x7f5d481a3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 0x7f5d481a3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 -- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 msgr2=0x7f5d481a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d481a3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d46d9d640 1 -- 172.21.15.32:0/3564302501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d481a80f0 con 0x7f5d48116fe0 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.814+0000 7f5d4659c640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 0x7f5d481a3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d46d9d640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009c90 tx=0x7f5d4000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40010630 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d40010c80 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d481a83e0 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40017c80 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.815+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d481afc70 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.816+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d48068e00 con 0x7f5d48116fe0 2024-04-23T18:26:31.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.817+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5d400107d0 con 0x7f5d48116fe0 2024-04-23T18:26:31.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.818+0000 7f5d377fe640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 0x7f5d300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:31.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.818+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f5d400d1390 con 0x7f5d48116fe0 2024-04-23T18:26:31.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.818+0000 7f5d4659c640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 0x7f5d300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:31.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.818+0000 7f5d4659c640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 0x7f5d300a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d38031440 tx=0x7f5d38005b50 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:31.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:31.822+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d4009da80 con 0x7f5d48116fe0 2024-04-23T18:26:31.920 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:31 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2248004073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:32.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:31 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2248004073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:32.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.208+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d481afed0 con 0x7f5d48116fe0 2024-04-23T18:26:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.209+0000 7f5d377fe640 1 -- 172.21.15.32:0/3564302501 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11855 (secure 0 0 0) 0x7f5d4009d380 con 0x7f5d48116fe0 2024-04-23T18:26:32.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:26:32.203 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:26:05.875467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:26:32.203 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-04-23T18:26:32.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 msgr2=0x7f5d300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 0x7f5d300a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d38031440 tx=0x7f5d38005b50 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 msgr2=0x7f5d481a79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009c90 tx=0x7f5d4000c6a0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 shutdown_connections 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d48116fe0 0x7f5d481a79c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5d300a3d20 0x7f5d300a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4810c6e0 0x7f5d481a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 --2- 172.21.15.32:0/3564302501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d48109490 0x7f5d481a3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.212+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 >> 172.21.15.32:0/3564302501 conn(0x7f5d4806e540 msgr2=0x7f5d48118820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.213+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 shutdown_connections 2024-04-23T18:26:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.213+0000 7f5d4cf22640 1 -- 172.21.15.32:0/3564302501 wait complete. 2024-04-23T18:26:32.222 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:26:32.222 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/1652089674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0108f50 msgr2=0x7f9eb0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/1652089674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0108f50 0x7f9eb0109330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009930 tx=0x7f9ea002a420 comp rx=0 tx=0).stop 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/1652089674 shutdown_connections 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/1652089674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb01150a0 0x7f9eb0117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/1652089674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb0109870 0x7f9eb0114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/1652089674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0108f50 0x7f9eb0109330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/1652089674 >> 172.21.15.32:0/1652089674 conn(0x7f9eb006e2e0 msgr2=0x7f9eb006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/1652089674 shutdown_connections 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/1652089674 wait complete. 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.386+0000 7f9eb5fa1640 1 Processor -- start 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 -- start start 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb0108f50 0x7f9eb01a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 0x7f9eb01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb011aa40 con 0x7f9eb01150a0 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb011a8c0 con 0x7f9eb0108f50 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eb5fa1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb011abc0 con 0x7f9eb0109870 2024-04-23T18:26:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 0x7f9eb01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39628/0 (socket says 172.21.15.32:39628) 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/923466118 learned_addr learned my addr 172.21.15.32:0/923466118 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 0x7f9eb01a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38036/0 (socket says 172.21.15.32:38036) 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.387+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb0108f50 0x7f9eb01a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/923466118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb0108f50 msgr2=0x7f9eb01a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb0108f50 0x7f9eb01a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 msgr2=0x7f9eb01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 0x7f9eb01a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/923466118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb01a81a0 con 0x7f9eb0109870 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800e9f0 tx=0x7f9e9800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ead7fa640 1 -- 172.21.15.32:0/923466118 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800cd80 con 0x7f9eb0109870 2024-04-23T18:26:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ead7fa640 1 -- 172.21.15.32:0/923466118 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e980108e0 con 0x7f9eb0109870 2024-04-23T18:26:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9ead7fa640 1 -- 172.21.15.32:0/923466118 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98018980 con 0x7f9eb0109870 2024-04-23T18:26:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb01a8490 con 0x7f9eb0109870 2024-04-23T18:26:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.388+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb01afd20 con 0x7f9eb0109870 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.390+0000 7f9ead7fa640 1 -- 172.21.15.32:0/923466118 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9e98010430 con 0x7f9eb0109870 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 Processor -- start 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 -- start start 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55728c849370 con 0x55728c8e49c0 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55728c7e62e0 con 0x55728c8ef090 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55728c845330 con 0x55728c8e40a0 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ead7fa640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 0x7f9e9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:56168/0 (socket says 172.21.15.32:56168) 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/2301886247 learned_addr learned my addr 172.21.15.32:0/2301886247 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:39636/0 (socket says 172.21.15.32:39636) 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaffff640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 0x7f9e9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 -- 172.21.15.32:0/2301886247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 msgr2=0x55728c8e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 -- 172.21.15.32:0/2301886247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 msgr2=0x55728c8f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaf7fe640 1 -- 172.21.15.32:0/2301886247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c627d80 con 0x55728c8e49c0 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ead7fa640 1 -- 172.21.15.32:0/923466118 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f9e980d0ac0 con 0x7f9eb0109870 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9eaffff640 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.391+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.392+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8eeb50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9ea002a940 tx=0x7f9ea00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c992a8dbd0d08133 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.392+0000 7f9eaffff640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 0x7f9e9c0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009fd0 tx=0x7f9ea8004ab0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.394+0000 7f9ea5ffb640 1 -- 172.21.15.32:0/2301886247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0035040 con 0x55728c8e49c0 2024-04-23T18:26:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.394+0000 7f9ea5ffb640 1 -- 172.21.15.32:0/2301886247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea00054f0 con 0x55728c8e49c0 2024-04-23T18:26:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.394+0000 7f9ea5ffb640 1 -- 172.21.15.32:0/2301886247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00035f0 con 0x55728c8e49c0 2024-04-23T18:26:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2301886247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 msgr2=0x55728c8eeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8eeb50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9ea002a940 tx=0x7f9ea00040c0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2301886247 shutdown_connections 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8eeb50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2301886247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2301886247 >> 172.21.15.32:0/2301886247 conn(0x55728c8e03c0 msgr2=0x55728c8e07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2301886247 shutdown_connections 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2301886247 wait complete. 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.395+0000 7f9eb8d8a740 1 -- start start 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 Processor -- start 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55728c8f1e20 con 0x55728c8e49c0 2024-04-23T18:26:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55728c7e3fb0 con 0x55728c8ef090 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55728c8f1fa0 con 0x55728c8e40a0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eb8d8a740 10 client.0 _ll_register_callbacks cb 0x55728c7e39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38062/0 (socket says 172.21.15.32:38062) 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/2842374628 learned_addr learned my addr 172.21.15.32:0/2842374628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eaffff640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/2842374628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 msgr2=0x55728c8ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.396+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 msgr2=0x55728c8ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8ecb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9ea6ffd640 1 -- 172.21.15.32:0/2842374628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c688510 con 0x55728c8e49c0 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9ea6ffd640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9e98000c00 tx=0x7f9e9801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc563d001227b206 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e980ce6b0 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f9e980a09a0 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ea00054f0 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9e9809cb30 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9eb8d8a740 10 client.16896 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55728c51b670 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e980d6030 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.397+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f9e98098880 con 0x55728c8e49c0 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 10 client.16896 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 10 client.16896 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 20 client.16896 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c907c80 0x55728c90a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 4 client.16896 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55728c856d90 con 0x55728c907c80 2024-04-23T18:26:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c907c80 0x55728c90a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c907c80 0x55728c90a0b0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc8fcbb556e5716 server_cookie=14e4b5cf93261390 in_seq=0 out_seq=0 2024-04-23T18:26:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.398+0000 7f9e8effd640 10 client.16896 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9e8effd640 1 -- 172.21.15.32:0/2842374628 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55728c856d90 con 0x55728c907c80 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9e8effd640 10 client.16896 handle_command_reply: tid=0 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 client.16896 shutdown 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 msgr2=0x55728c8ec5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9e98000c00 tx=0x7f9e9801f000 comp rx=0 tx=0).stop 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 shutdown_connections 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c907c80 0x55728c90a0b0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8ef090 0x55728c8ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e49c0 0x55728c8ec5d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9e8effd640 0 client.16896 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/2842374628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 >> 172.21.15.32:0/2842374628 conn(0x55728c8e03c0 msgr2=0x55728c8e61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 shutdown_connections 2024-04-23T18:26:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.399+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/2842374628 wait complete. 2024-04-23T18:26:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.400+0000 7f9eb8d8a740 20 client.16896 trim_cache size 0 max 16384 2024-04-23T18:26:32.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.422+0000 7f9eb8d8a740 1 Processor -- start 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.422+0000 7f9eb8d8a740 1 -- start start 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.422+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.422+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c90f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.422+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 0x55728c90fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b6c20 con 0x55728c8e40a0 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b4460 con 0x55728c8e49c0 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b22f0 con 0x55728c8ef090 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38072/0 (socket says 172.21.15.32:38072) 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 0x55728c90fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaf7fe640 1 -- 172.21.15.32:0/4268906027 learned_addr learned my addr 172.21.15.32:0/4268906027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9ea6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c90f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 -- 172.21.15.32:0/4268906027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 msgr2=0x55728c90f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c90f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 -- 172.21.15.32:0/4268906027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 msgr2=0x55728c903570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 -- 172.21.15.32:0/4268906027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c8e36b0 con 0x55728c8ef090 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9eaffff640 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 0x55728c90fec0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8005fd0 tx=0x7f9ea8009c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d67b0f97f890b66e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9ea57fa640 1 -- 172.21.15.32:0/4268906027 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8022b50 con 0x55728c8ef090 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9ea57fa640 1 -- 172.21.15.32:0/4268906027 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea800fe40 con 0x55728c8ef090 2024-04-23T18:26:32.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.423+0000 7f9ea57fa640 1 -- 172.21.15.32:0/4268906027 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea802a3e0 con 0x55728c8ef090 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/4268906027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 msgr2=0x55728c90fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 0x55728c90fec0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8005fd0 tx=0x7f9ea8009c80 comp rx=0 tx=0).stop 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/4268906027 shutdown_connections 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8ef090 0x55728c90fec0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c90f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/4268906027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8e40a0 0x55728c903570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/4268906027 >> 172.21.15.32:0/4268906027 conn(0x55728c8e03c0 msgr2=0x55728c8e5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/4268906027 shutdown_connections 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.424+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/4268906027 wait complete. 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.425+0000 7f9eb8d8a740 1 -- start start 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.425+0000 7f9eb8d8a740 1 Processor -- start 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c8fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b2940 con 0x55728c8ef090 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b3d90 con 0x55728c8e49c0 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b2ff0 con 0x55728c8e40a0 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eb8d8a740 10 client.0 _ll_register_callbacks cb 0x7f9e98009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eaffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38074/0 (socket says 172.21.15.32:38074) 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.426+0000 7f9eaffff640 1 -- 172.21.15.32:0/3521863883 learned_addr learned my addr 172.21.15.32:0/3521863883 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 -- 172.21.15.32:0/3521863883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 msgr2=0x55728c8fbb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8fbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 -- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 msgr2=0x55728c8fc0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c8fc0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 -- 172.21.15.32:0/3521863883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c96d730 con 0x55728c8ef090 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9eaffff640 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9ea802b000 tx=0x7f9ea8010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14144c6f30fb7cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9e8effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.427+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8039060 con 0x55728c8ef090 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.428+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f9ea8022c20 con 0x55728c8ef090 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.428+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55728c850c00 con 0x55728c8ef090 2024-04-23T18:26:32.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.428+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9ea8022e20 con 0x55728c8ef090 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.428+0000 7f9eb8d8a740 10 client.16902 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.428+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55728c96e9c0 con 0x55728c8ef090 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea8003e20 con 0x55728c8ef090 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f9ea8029070 con 0x55728c8ef090 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 10 client.16902 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 10 client.16902 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 20 client.16902 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c96f080 0x55728c9714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 4 client.16902 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55728c850c00 con 0x55728c96f080 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c96f080 0x55728c9714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9e8effd640 10 client.16902 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.429+0000 7f9eaf7fe640 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c96f080 0x55728c9714a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23b7f4b3f5657ff8 server_cookie=b20da1b2f1b21a92 in_seq=0 out_seq=0 2024-04-23T18:26:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.658+0000 7f9e8effd640 1 -- 172.21.15.32:0/3521863883 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55728c850c00 con 0x55728c96f080 2024-04-23T18:26:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.658+0000 7f9e8effd640 10 client.16902 handle_command_reply: tid=0 2024-04-23T18:26:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.658+0000 7f9eb8d8a740 1 client.16902 shutdown 2024-04-23T18:26:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 msgr2=0x55728c96d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9ea802b000 tx=0x7f9ea8010f60 comp rx=0 tx=0).stop 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 shutdown_connections 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55728c96f080 0x55728c9714a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55728c8ef090 0x55728c96d1f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55728c8e49c0 0x55728c8fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 --2- 172.21.15.32:0/3521863883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55728c8e40a0 0x55728c8fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9e8effd640 0 client.16902 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 >> 172.21.15.32:0/3521863883 conn(0x55728c8e03c0 msgr2=0x55728c8df600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 shutdown_connections 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 1 -- 172.21.15.32:0/3521863883 wait complete. 2024-04-23T18:26:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb8d8a740 20 client.16902 trim_cache size 0 max 16384 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 42, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 42, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "198bd97f": { 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 42, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:26:32.654 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:26:32.655 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:26:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 msgr2=0x7f9e9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 0x7f9e9c0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009fd0 tx=0x7f9ea8004ab0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 msgr2=0x7f9eb01a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.659+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800e9f0 tx=0x7f9e9800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 shutdown_connections 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e9c0a3d20 0x7f9e9c0a61e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb01150a0 0x7f9eb01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eb0109870 0x7f9eb01a3740 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 --2- 172.21.15.32:0/923466118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eb0108f50 0x7f9eb01a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 >> 172.21.15.32:0/923466118 conn(0x7f9eb006e2e0 msgr2=0x7f9eb010a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 shutdown_connections 2024-04-23T18:26:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:26:32.660+0000 7f9eb5fa1640 1 -- 172.21.15.32:0/923466118 wait complete. 2024-04-23T18:26:32.663 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '198bd97f', quiesce took 1.0 seconds. Will release after: 30.0 2024-04-23T18:26:32.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:32 smithi064 ceph-mon[33062]: pgmap v356: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-23T18:26:32.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:32 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3564302501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:33.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:32 smithi032 ceph-mon[29221]: pgmap v356: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-23T18:26:33.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:32 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3564302501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:32 smithi040 ceph-mon[32626]: pgmap v356: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-23T18:26:33.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:32 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3564302501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:33.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 665.0 s, 225.6 tps, lat 2809.834 ms stddev 8784.266 2024-04-23T18:26:34.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:34 smithi064 ceph-mon[33062]: pgmap v357: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-04-23T18:26:35.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:34 smithi032 ceph-mon[29221]: pgmap v357: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-04-23T18:26:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:34 smithi040 ceph-mon[32626]: pgmap v357: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 33 KiB/s wr, 13 op/s 2024-04-23T18:26:36.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:36 smithi064 ceph-mon[33062]: pgmap v358: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 507 KiB/s wr, 63 op/s 2024-04-23T18:26:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:36 smithi032 ceph-mon[29221]: pgmap v358: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 507 KiB/s wr, 63 op/s 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 -- 172.21.15.32:0/3207346024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64106220 msgr2=0x7f5b6410dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 --2- 172.21.15.32:0/3207346024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64106220 0x7f5b6410dda0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009cd0 tx=0x7f5b5c02a8d0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 -- 172.21.15.32:0/3207346024 shutdown_connections 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 --2- 172.21.15.32:0/3207346024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64106220 0x7f5b6410dda0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 --2- 172.21.15.32:0/3207346024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64101d00 0x7f5b64105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 --2- 172.21.15.32:0/3207346024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b64101350 0x7f5b64101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 -- 172.21.15.32:0/3207346024 >> 172.21.15.32:0/3207346024 conn(0x7f5b64077d10 msgr2=0x7f5b64078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 -- 172.21.15.32:0/3207346024 shutdown_connections 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.051+0000 7f5b6a322640 1 -- 172.21.15.32:0/3207346024 wait complete. 2024-04-23T18:26:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 Processor -- start 2024-04-23T18:26:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 -- start start 2024-04-23T18:26:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64106220 0x7f5b64198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 0x7f5b6419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b64111100 con 0x7f5b64101350 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b64110f80 con 0x7f5b6419cfd0 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b6a322640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5b64111280 con 0x7f5b64106220 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.052+0000 7f5b63fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37872/0 (socket says 172.21.15.32:37872) 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 -- 172.21.15.32:0/938076043 learned_addr learned my addr 172.21.15.32:0/938076043 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b637fe640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64106220 0x7f5b64198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b688bf640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 0x7f5b6419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 -- 172.21.15.32:0/938076043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64106220 msgr2=0x7f5b64198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64106220 0x7f5b64198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 -- 172.21.15.32:0/938076043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 msgr2=0x7f5b6419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 0x7f5b6419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 -- 172.21.15.32:0/938076043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6419dac0 con 0x7f5b64101350 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b688bf640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 0x7f5b6419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b63fff640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d950 tx=0x7f5b5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b617fa640 1 -- 172.21.15.32:0/938076043 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54017070 con 0x7f5b64101350 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6419ddb0 con 0x7f5b64101350 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b617fa640 1 -- 172.21.15.32:0/938076043 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b54009c30 con 0x7f5b64101350 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b617fa640 1 -- 172.21.15.32:0/938076043 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400f620 con 0x7f5b64101350 2024-04-23T18:26:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.053+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b641a55a0 con 0x7f5b64101350 2024-04-23T18:26:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 Processor -- start 2024-04-23T18:26:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b617fa640 1 -- 172.21.15.32:0/938076043 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5b54010460 con 0x7f5b64101350 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 -- start start 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef2c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef2ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a2aee86970 con 0x55a2aef21620 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a2aee234a0 con 0x55a2aef2c800 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a2aee82930 con 0x55a2aef21f40 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b688bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b688bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37888/0 (socket says 172.21.15.32:37888) 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b688bf640 1 -- 172.21.15.32:0/1966477831 learned_addr learned my addr 172.21.15.32:0/1966477831 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef2c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b637fe640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef2ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.056+0000 7f5b617fa640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 0x7f5b4c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 -- 172.21.15.32:0/1966477831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 msgr2=0x55a2aef2c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef2c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 -- 172.21.15.32:0/1966477831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 msgr2=0x55a2aef2ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef2ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 -- 172.21.15.32:0/1966477831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2aeb99210 con 0x55a2aef21620 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b617fa640 1 -- 172.21.15.32:0/938076043 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f5b540d0420 con 0x7f5b64101350 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 0x7f5b4c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b688bf640 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009a60 tx=0x7f5b5c031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeffd74f8126fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b4a7fc640 1 -- 172.21.15.32:0/1966477831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c030b20 con 0x55a2aef21620 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b4a7fc640 1 -- 172.21.15.32:0/1966477831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5c036070 con 0x55a2aef21620 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.057+0000 7f5b4a7fc640 1 -- 172.21.15.32:0/1966477831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c046d40 con 0x55a2aef21620 2024-04-23T18:26:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1966477831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 msgr2=0x55a2aef21a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009a60 tx=0x7f5b5c031a30 comp rx=0 tx=0).stop 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b688bf640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 0x7f5b4c0a61d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c04a000 tx=0x7f5b5c047000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1966477831 shutdown_connections 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef2ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1966477831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21620 0x55a2aef21a00 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1966477831 >> 172.21.15.32:0/1966477831 conn(0x55a2aef1db30 msgr2=0x55a2aef1df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1966477831 shutdown_connections 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1966477831 wait complete. 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.058+0000 7f5b6d03f740 1 -- start start 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 Processor -- start 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 0x55a2aef2c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef299a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a2aef2f430 con 0x55a2aef21f40 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a2aee210b0 con 0x55a2aef2c800 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a2aef2f5b0 con 0x55a2aef21620 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b6d03f740 10 client.0 _ll_register_callbacks cb 0x55a2aee0ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37900/0 (socket says 172.21.15.32:37900) 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b637fe640 1 -- 172.21.15.32:0/1702756632 learned_addr learned my addr 172.21.15.32:0/1702756632 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b688bf640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.059+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 0x55a2aef2c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 -- 172.21.15.32:0/1702756632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 msgr2=0x55a2aef2c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 0x55a2aef2c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 -- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 msgr2=0x55a2aef299a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef299a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 -- 172.21.15.32:0/1702756632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2aec65620 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 0x55a2aef2c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b637fe640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f5b580126e0 tx=0x7f5b58012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce38d034af1f9fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58004020 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f5b580041c0 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b5c036070 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f5b580043c0 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b6d03f740 10 client.16914 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a2aef3b410 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.060+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5c036070 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f5b58003bc0 con 0x55a2aef21f40 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 10 client.16914 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 10 client.16914 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 20 client.16914 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aef3c920 0x55a2aef3ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 4 client.16914 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2aee94390 con 0x55a2aef3c920 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aef3c920 0x55a2aef3ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b63fff640 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aef3c920 0x55a2aef3ed30 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cf19be6209f7a7a server_cookie=d68fc150157d3db3 in_seq=0 out_seq=0 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.061+0000 7f5b437fe640 10 client.16914 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b437fe640 1 -- 172.21.15.32:0/1702756632 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2aee94390 con 0x55a2aef3c920 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b437fe640 10 client.16914 handle_command_reply: tid=0 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 client.16914 shutdown 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 msgr2=0x55a2aef29440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f5b580126e0 tx=0x7f5b58012ba0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 shutdown_connections 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aef3c920 0x55a2aef3ed30 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef2c800 0x55a2aef299a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef29440 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/1702756632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21620 0x55a2aef2c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b437fe640 0 client.16914 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 >> 172.21.15.32:0/1702756632 conn(0x55a2aef1db30 msgr2=0x55a2aef23860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 shutdown_connections 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 1 -- 172.21.15.32:0/1702756632 wait complete. 2024-04-23T18:26:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.062+0000 7f5b6d03f740 20 client.16914 trim_cache size 0 max 16384 2024-04-23T18:26:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.077+0000 7f5b6d03f740 1 Processor -- start 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 -- start start 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 0x55a2aef44640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641b0500 con 0x55a2aef21f40 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641add40 con 0x55a2aef21620 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641abbb0 con 0x55a2aef2c800 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37912/0 (socket says 172.21.15.32:37912) 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 -- 172.21.15.32:0/591145690 learned_addr learned my addr 172.21.15.32:0/591145690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b63fff640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 -- 172.21.15.32:0/591145690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 msgr2=0x55a2aef44640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b688bf640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 0x55a2aef44640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 0x55a2aef44640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 -- 172.21.15.32:0/591145690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 msgr2=0x55a2aef382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 -- 172.21.15.32:0/591145690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2aef20e20 con 0x55a2aef21f40 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b63fff640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b688bf640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 0x55a2aef44640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.078+0000 7f5b637fe640 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5b580059f0 tx=0x7f5b5800f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62512463f17068 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b49ffb640 1 -- 172.21.15.32:0/591145690 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58029030 con 0x55a2aef21f40 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b49ffb640 1 -- 172.21.15.32:0/591145690 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5801c6b0 con 0x55a2aef21f40 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b49ffb640 1 -- 172.21.15.32:0/591145690 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800fae0 con 0x55a2aef21f40 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 -- 172.21.15.32:0/591145690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 msgr2=0x55a2aef2bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5b580059f0 tx=0x7f5b5800f840 comp rx=0 tx=0).stop 2024-04-23T18:26:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 -- 172.21.15.32:0/591145690 shutdown_connections 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef2c800 0x55a2aef44640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef21f40 0x55a2aef2bc80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/591145690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 -- 172.21.15.32:0/591145690 >> 172.21.15.32:0/591145690 conn(0x55a2aef1db30 msgr2=0x55a2aef1ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 -- 172.21.15.32:0/591145690 shutdown_connections 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.079+0000 7f5b6d03f740 1 -- 172.21.15.32:0/591145690 wait complete. 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 -- start start 2024-04-23T18:26:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 Processor -- start 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef263f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef26930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641ac200 con 0x55a2aef2c800 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641ad670 con 0x55a2aef21620 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641ac8d0 con 0x55a2aef21f40 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b6d03f740 10 client.0 _ll_register_callbacks cb 0x7f5b58009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b688bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b688bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37918/0 (socket says 172.21.15.32:37918) 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b688bf640 1 -- 172.21.15.32:0/2148776460 learned_addr learned my addr 172.21.15.32:0/2148776460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b63fff640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef263f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.080+0000 7f5b637fe640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef26930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 -- 172.21.15.32:0/2148776460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 msgr2=0x55a2aef26930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef26930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 -- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 msgr2=0x55a2aef263f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef263f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 -- 172.21.15.32:0/2148776460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2aefa2100 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b688bf640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c02adf0 tx=0x7f5b5c031970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e6e30170eaeeb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c056040 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f5b5c030b20 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b5801c6b0 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f5b5c068050 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 10 client.16920 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a2aefa3380 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5801c6b0 con 0x55a2aef2c800 2024-04-23T18:26:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f5b5c059070 con 0x55a2aef2c800 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 10 client.16920 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 10 client.16920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 20 client.16920 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.081+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aefa3760 0x55a2aefa5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.082+0000 7f5b6d03f740 4 client.16920 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.082+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2aee8e200 con 0x55a2aefa3760 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.082+0000 7f5b63fff640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aefa3760 0x55a2aefa5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.082+0000 7f5b437fe640 10 client.16920 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.082+0000 7f5b63fff640 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aefa3760 0x55a2aefa5b50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2561bf19530bb1de server_cookie=6ba2b14dddc3e133 in_seq=0 out_seq=0 2024-04-23T18:26:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b437fe640 1 -- 172.21.15.32:0/2148776460 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2aee8e200 con 0x55a2aefa3760 2024-04-23T18:26:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b437fe640 10 client.16920 handle_command_reply: tid=0 2024-04-23T18:26:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 client.16920 shutdown 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 msgr2=0x55a2aef26e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c02adf0 tx=0x7f5b5c031970 comp rx=0 tx=0).stop 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 shutdown_connections 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a2aefa3760 0x55a2aefa5b50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2aef2c800 0x55a2aef26e70 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2aef21f40 0x55a2aef26930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b437fe640 0 client.16920 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 --2- 172.21.15.32:0/2148776460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a2aef21620 0x55a2aef263f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 >> 172.21.15.32:0/2148776460 conn(0x55a2aef1db30 msgr2=0x55a2aef24140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 shutdown_connections 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 1 -- 172.21.15.32:0/2148776460 wait complete. 2024-04-23T18:26:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.097+0000 7f5b6d03f740 20 client.16920 trim_cache size 0 max 16384 2024-04-23T18:26:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:26:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:26:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:26:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 msgr2=0x7f5b4c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 0x7f5b4c0a61d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c04a000 tx=0x7f5b5c047000 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 msgr2=0x7f5b64198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d950 tx=0x7f5b5400de10 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 shutdown_connections 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b6419cfd0 0x7f5b6419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b64106220 0x7f5b64198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5b4c0a3d10 0x7f5b4c0a61d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 --2- 172.21.15.32:0/938076043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b64101350 0x7f5b64198710 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 >> 172.21.15.32:0/938076043 conn(0x7f5b64077d10 msgr2=0x7f5b6406a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 shutdown_connections 2024-04-23T18:26:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:37.098+0000 7f5b6a322640 1 -- 172.21.15.32:0/938076043 wait complete. 2024-04-23T18:26:37.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:36 smithi040 ceph-mon[32626]: pgmap v358: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 507 KiB/s wr, 63 op/s 2024-04-23T18:26:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 670.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:38.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:38 smithi064 ceph-mon[33062]: pgmap v359: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 504 KiB/s wr, 58 op/s 2024-04-23T18:26:39.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:38 smithi032 ceph-mon[29221]: pgmap v359: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 504 KiB/s wr, 58 op/s 2024-04-23T18:26:39.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:38 smithi040 ceph-mon[32626]: pgmap v359: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 504 KiB/s wr, 58 op/s 2024-04-23T18:26:40.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:40 smithi064 ceph-mon[33062]: pgmap v360: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 518 KiB/s wr, 64 op/s 2024-04-23T18:26:41.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:40 smithi032 ceph-mon[29221]: pgmap v360: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 518 KiB/s wr, 64 op/s 2024-04-23T18:26:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:40 smithi040 ceph-mon[32626]: pgmap v360: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 518 KiB/s wr, 64 op/s 2024-04-23T18:26:42.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:42 smithi064 ceph-mon[33062]: pgmap v361: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:43.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:42 smithi032 ceph-mon[29221]: pgmap v361: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:43.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:42 smithi040 ceph-mon[32626]: pgmap v361: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 675.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:44.918 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:44 smithi040 ceph-mon[32626]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:44.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:44 smithi064 ceph-mon[33062]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:45.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:44 smithi032 ceph-mon[29221]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 501 KiB/s wr, 58 op/s 2024-04-23T18:26:46.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:46 smithi064 ceph-mon[33062]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 511 KiB/s wr, 61 op/s 2024-04-23T18:26:47.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:46 smithi032 ceph-mon[29221]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 511 KiB/s wr, 61 op/s 2024-04-23T18:26:47.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:46 smithi040 ceph-mon[32626]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 511 KiB/s wr, 61 op/s 2024-04-23T18:26:48.865 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 680.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:48.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:48 smithi064 ceph-mon[33062]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 KiB/s wr, 11 op/s 2024-04-23T18:26:49.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:48 smithi032 ceph-mon[29221]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 KiB/s wr, 11 op/s 2024-04-23T18:26:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:48 smithi040 ceph-mon[32626]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 KiB/s wr, 11 op/s 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 -- 172.21.15.32:0/703617667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101ad0 msgr2=0x7fef10105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 --2- 172.21.15.32:0/703617667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101ad0 0x7fef10105ac0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7feef8009900 tx=0x7feef802a410 comp rx=0 tx=0).stop 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 -- 172.21.15.32:0/703617667 shutdown_connections 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 --2- 172.21.15.32:0/703617667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101061a0 0x7fef1010dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 --2- 172.21.15.32:0/703617667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101ad0 0x7fef10105ac0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 --2- 172.21.15.32:0/703617667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef10101120 0x7fef10101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 -- 172.21.15.32:0/703617667 >> 172.21.15.32:0/703617667 conn(0x7fef10077fa0 msgr2=0x7fef100783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 -- 172.21.15.32:0/703617667 shutdown_connections 2024-04-23T18:26:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.187+0000 7fef151cd640 1 -- 172.21.15.32:0/703617667 wait complete. 2024-04-23T18:26:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 Processor -- start 2024-04-23T18:26:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 -- start start 2024-04-23T18:26:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef101061a0 0x7fef101a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef1019a780 0x7fef1019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fef10111130 con 0x7fef10101120 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef10110fb0 con 0x7fef101061a0 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef151cd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef101112b0 con 0x7fef1019a780 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.188+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48244/0 (socket says 172.21.15.32:48244) 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1425916375 learned_addr learned my addr 172.21.15.32:0/1425916375 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0f59e640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef1019a780 0x7fef1019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1425916375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef1019a780 msgr2=0x7fef1019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef07fff640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef101061a0 0x7fef101a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef1019a780 0x7fef1019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1425916375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef101061a0 msgr2=0x7fef101a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef101061a0 0x7fef101a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1425916375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1019b310 con 0x7fef10101120 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fef0000d950 tx=0x7fef0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0cd99640 1 -- 172.21.15.32:0/1425916375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00017030 con 0x7fef10101120 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1019b600 con 0x7fef10101120 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0cd99640 1 -- 172.21.15.32:0/1425916375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef00004020 con 0x7fef10101120 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef0cd99640 1 -- 172.21.15.32:0/1425916375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000047a0 con 0x7fef10101120 2024-04-23T18:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.189+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef101a77e0 con 0x7fef10101120 2024-04-23T18:26:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.191+0000 7fef0cd99640 1 -- 172.21.15.32:0/1425916375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fef00009a30 con 0x7fef10101120 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 Processor -- start 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 -- start start 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x5614859fca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a09bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561485961970 con 0x5614859fcf40 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5614858fe4a0 con 0x561485a07800 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef17fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56148595d930 con 0x5614859fc620 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef0cd99640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 0x7feefc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef0f59e640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 0x7feefc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48258/0 (socket says 172.21.15.32:48258) 2024-04-23T18:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1231293030 learned_addr learned my addr 172.21.15.32:0/1231293030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.192+0000 7fef07fff640 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a09bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0cd99640 1 -- 172.21.15.32:0/1425916375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fef000d02f0 con 0x7fef10101120 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0f59e640 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x5614859fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1231293030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 msgr2=0x5614859fca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x5614859fca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1231293030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 msgr2=0x561485a09bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a09bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 -- 172.21.15.32:0/1231293030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561485674210 con 0x5614859fcf40 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0f59e640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 0x7feefc0a61d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef08005c40 tx=0x7fef08009c90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fef0009bfb0 tx=0x7fef0006eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa00c5d70978f0b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef057fa640 1 -- 172.21.15.32:0/1231293030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0007e1c0 con 0x5614859fcf40 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef057fa640 1 -- 172.21.15.32:0/1231293030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef00066dc0 con 0x5614859fcf40 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.193+0000 7fef057fa640 1 -- 172.21.15.32:0/1231293030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000cdce0 con 0x5614859fcf40 2024-04-23T18:26:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- 172.21.15.32:0/1231293030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 msgr2=0x561485a072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fef0009bfb0 tx=0x7fef0006eda0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- 172.21.15.32:0/1231293030 shutdown_connections 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a09bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x561485a072c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 --2- 172.21.15.32:0/1231293030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x5614859fca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- 172.21.15.32:0/1231293030 >> 172.21.15.32:0/1231293030 conn(0x5614859f8b30 msgr2=0x5614859f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- 172.21.15.32:0/1231293030 shutdown_connections 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- 172.21.15.32:0/1231293030 wait complete. 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 -- start start 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.194+0000 7fef17fbe740 1 Processor -- start 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614859fc620 0x561485a07200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561485a0a3d0 con 0x561485a07800 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5614858fc0b0 con 0x5614859fc620 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561485a0a550 con 0x5614859fcf40 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 10 client.0 _ll_register_callbacks cb 0x5614858e9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48270/0 (socket says 172.21.15.32:48270) 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 -- 172.21.15.32:0/2396627796 learned_addr learned my addr 172.21.15.32:0/2396627796 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef07fff640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614859fc620 0x561485a07200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 -- 172.21.15.32:0/2396627796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 msgr2=0x561485a044f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a044f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 -- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614859fc620 msgr2=0x561485a07200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614859fc620 0x561485a07200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 -- 172.21.15.32:0/2396627796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561485740620 con 0x561485a07800 2024-04-23T18:26:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef0f59e640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fef08010c60 tx=0x7fef0801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295938e71be9557c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7feef67fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08027070 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fef0801ee00 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.195+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef00066dc0 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fef08033050 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 10 client.16932 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561485a16380 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef00066dc0 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7fef0801f070 con 0x561485a07800 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 10 client.16932 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 10 client.16932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 20 client.16932 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a178b0 0x561485a19cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 4 client.16932 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56148596f390 con 0x561485a178b0 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.196+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a178b0 0x561485a19cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.197+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a178b0 0x561485a19cc0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af9861af3c0b8ae4 server_cookie=411f5329da9d8fe4 in_seq=0 out_seq=0 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.197+0000 7feef67fc640 10 client.16932 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.197+0000 7feef67fc640 1 -- 172.21.15.32:0/2396627796 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56148596f390 con 0x561485a178b0 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.197+0000 7feef67fc640 10 client.16932 handle_command_reply: tid=0 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.197+0000 7fef17fbe740 1 client.16932 shutdown 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 msgr2=0x561485a04b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fef08010c60 tx=0x7fef0801c000 comp rx=0 tx=0).stop 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 shutdown_connections 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a178b0 0x561485a19cc0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561485a07800 0x561485a04b00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a044f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7feef67fc640 0 client.16932 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 --2- 172.21.15.32:0/2396627796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614859fc620 0x561485a07200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 >> 172.21.15.32:0/2396627796 conn(0x5614859f8b30 msgr2=0x5614859fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 shutdown_connections 2024-04-23T18:26:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 1 -- 172.21.15.32:0/2396627796 wait complete. 2024-04-23T18:26:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.198+0000 7fef17fbe740 20 client.16932 trim_cache size 0 max 16384 2024-04-23T18:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.212+0000 7fef17fbe740 1 Processor -- start 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.212+0000 7fef17fbe740 1 -- start start 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x561485a13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a1f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fef101b2740 con 0x5614859fcf40 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef101aff80 con 0x561485a07800 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef17fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef101addf0 con 0x5614859fc620 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48274/0 (socket says 172.21.15.32:48274) 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 -- 172.21.15.32:0/257045755 learned_addr learned my addr 172.21.15.32:0/257045755 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef0f59e640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a1f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 -- 172.21.15.32:0/257045755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 msgr2=0x561485a13290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x561485a13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x561485a13290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 -- 172.21.15.32:0/257045755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 msgr2=0x561485a1f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a1f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 -- 172.21.15.32:0/257045755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614859fcd70 con 0x5614859fcf40 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x561485a13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.213+0000 7fef07fff640 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7feef8009230 tx=0x7feef8004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb63c9a63b24ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef04ff9640 1 -- 172.21.15.32:0/257045755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8035040 con 0x5614859fcf40 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef04ff9640 1 -- 172.21.15.32:0/257045755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8041070 con 0x5614859fcf40 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef04ff9640 1 -- 172.21.15.32:0/257045755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef80035f0 con 0x5614859fcf40 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 -- 172.21.15.32:0/257045755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 msgr2=0x5614859f7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7feef8009230 tx=0x7feef8004280 comp rx=0 tx=0).stop 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 -- 172.21.15.32:0/257045755 shutdown_connections 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a1f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fcf40 0x5614859f7b30 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 --2- 172.21.15.32:0/257045755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fc620 0x561485a13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 -- 172.21.15.32:0/257045755 >> 172.21.15.32:0/257045755 conn(0x5614859f8b30 msgr2=0x5614859f7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 -- 172.21.15.32:0/257045755 shutdown_connections 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.214+0000 7fef17fbe740 1 -- 172.21.15.32:0/257045755 wait complete. 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 -- start start 2024-04-23T18:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 Processor -- start 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a0d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a0ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fef101ae440 con 0x5614859fc620 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef101af8b0 con 0x561485a07800 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef101aeb10 con 0x5614859fcf40 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef17fbe740 10 client.0 _ll_register_callbacks cb 0x7fef08009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48282/0 (socket says 172.21.15.32:48282) 2024-04-23T18:26:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 -- 172.21.15.32:0/3656673711 learned_addr learned my addr 172.21.15.32:0/3656673711 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef07fff640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a0d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0f59e640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a0ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 -- 172.21.15.32:0/3656673711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 msgr2=0x561485a0d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a0d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 -- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 msgr2=0x561485a0ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a0ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0ed9d640 1 -- 172.21.15.32:0/3656673711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561485a7cfb0 con 0x5614859fc620 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef07fff640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.215+0000 7fef0f59e640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a0ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef0ed9d640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fef000cdb10 tx=0x7fef0009bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f3d0903e3596f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0009c2b0 con 0x5614859fc620 2024-04-23T18:26:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fef0009a5b0 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef8041070 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fef00009d20 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 10 client.16938 fetch_fsmap learned FSMap version 42 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561485a7e210 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8041070 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7fef00098920 con 0x5614859fc620 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 10 client.16938 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 10 client.16938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 20 client.16938 populate_metadata read hostname 'smithi032' 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a7e550 0x561485a80940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 4 client.16938 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.216+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561485969200 con 0x561485a7e550 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.217+0000 7fef07fff640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a7e550 0x561485a80940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.217+0000 7fef07fff640 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a7e550 0x561485a80940 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f4ea9190d46b8b4 server_cookie=f4bc2af288624ec5 in_seq=0 out_seq=0 2024-04-23T18:26:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.217+0000 7feef67fc640 10 client.16938 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7feef67fc640 1 -- 172.21.15.32:0/3656673711 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561485969200 con 0x561485a7e550 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7feef67fc640 10 client.16938 handle_command_reply: tid=0 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 client.16938 shutdown 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 msgr2=0x561485a0d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fef000cdb10 tx=0x7fef0009bdf0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 shutdown_connections 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561485a7e550 0x561485a80940 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561485a07800 0x561485a0ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5614859fcf40 0x561485a0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7feef67fc640 0 client.16938 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 --2- 172.21.15.32:0/3656673711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614859fc620 0x561485a0d330 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 >> 172.21.15.32:0/3656673711 conn(0x5614859f8b30 msgr2=0x5614859f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 shutdown_connections 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 1 -- 172.21.15.32:0/3656673711 wait complete. 2024-04-23T18:26:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.232+0000 7fef17fbe740 20 client.16938 trim_cache size 0 max 16384 2024-04-23T18:26:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:26:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:26:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:26:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 msgr2=0x7feefc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 0x7feefc0a61d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef08005c40 tx=0x7fef08009c90 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 msgr2=0x7fef101a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fef0000d950 tx=0x7fef0000de10 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 shutdown_connections 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef1019a780 0x7fef1019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef101061a0 0x7fef101a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7feefc0a3d10 0x7feefc0a61d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 --2- 172.21.15.32:0/1425916375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef10101120 0x7fef101a0610 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 >> 172.21.15.32:0/1425916375 conn(0x7fef10077fa0 msgr2=0x7fef100ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 shutdown_connections 2024-04-23T18:26:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:26:50.233+0000 7fef151cd640 1 -- 172.21.15.32:0/1425916375 wait complete. 2024-04-23T18:26:50.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:50 smithi064 ceph-mon[33062]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-04-23T18:26:51.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:50 smithi032 ceph-mon[29221]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-04-23T18:26:51.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:50 smithi040 ceph-mon[32626]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 45 KiB/s wr, 16 op/s 2024-04-23T18:26:52.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:52 smithi064 ceph-mon[33062]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 31 KiB/s wr, 10 op/s 2024-04-23T18:26:53.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:52 smithi032 ceph-mon[29221]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 31 KiB/s wr, 10 op/s 2024-04-23T18:26:53.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:52 smithi040 ceph-mon[32626]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 31 KiB/s wr, 10 op/s 2024-04-23T18:26:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 685.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:54.921 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:54 smithi064 ceph-mon[33062]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-23T18:26:55.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:54 smithi032 ceph-mon[29221]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-23T18:26:55.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:54 smithi040 ceph-mon[32626]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-23T18:26:57.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:56 smithi032 ceph-mon[29221]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 560 KiB/s wr, 78 op/s 2024-04-23T18:26:57.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:56 smithi040 ceph-mon[32626]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 560 KiB/s wr, 78 op/s 2024-04-23T18:26:57.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:56 smithi064 ceph-mon[33062]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 560 KiB/s wr, 78 op/s 2024-04-23T18:26:58.862 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 690.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:26:59.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:26:58 smithi032 ceph-mon[29221]: pgmap v369: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 538 KiB/s wr, 73 op/s 2024-04-23T18:26:59.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:26:58 smithi040 ceph-mon[32626]: pgmap v369: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 538 KiB/s wr, 73 op/s 2024-04-23T18:26:59.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:26:58 smithi064 ceph-mon[33062]: pgmap v369: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 538 KiB/s wr, 73 op/s 2024-04-23T18:27:01.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:00 smithi032 ceph-mon[29221]: pgmap v370: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 540 KiB/s wr, 77 op/s 2024-04-23T18:27:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:00 smithi040 ceph-mon[32626]: pgmap v370: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 540 KiB/s wr, 77 op/s 2024-04-23T18:27:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:00 smithi064 ceph-mon[33062]: pgmap v370: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 540 KiB/s wr, 77 op/s 2024-04-23T18:27:02.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 -- 172.21.15.32:0/2565275213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c101a50 msgr2=0x7fad0c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 --2- 172.21.15.32:0/2565275213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c101a50 0x7fad0c105a90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fad00009900 tx=0x7fad0002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 -- 172.21.15.32:0/2565275213 shutdown_connections 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 --2- 172.21.15.32:0/2565275213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c106170 0x7fad0c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 --2- 172.21.15.32:0/2565275213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c101a50 0x7fad0c105a90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 --2- 172.21.15.32:0/2565275213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c1010a0 0x7fad0c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 -- 172.21.15.32:0/2565275213 >> 172.21.15.32:0/2565275213 conn(0x7fad0c077f50 msgr2=0x7fad0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 -- 172.21.15.32:0/2565275213 shutdown_connections 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.789+0000 7fad13b20640 1 -- 172.21.15.32:0/2565275213 wait complete. 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 Processor -- start 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 -- start start 2024-04-23T18:27:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 0x7fad0c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 0x7fad0c1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c1110e0 con 0x7fad0c1010a0 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c110f60 con 0x7fad0c19a050 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad13b20640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c111260 con 0x7fad0c106170 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad118bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad118bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49824/0 (socket says 172.21.15.32:49824) 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad118bc640 1 -- 172.21.15.32:0/2652830664 learned_addr learned my addr 172.21.15.32:0/2652830664 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad110bb640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 0x7fad0c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.790+0000 7fad120bd640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 0x7fad0c1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 -- 172.21.15.32:0/2652830664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 msgr2=0x7fad0c199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 0x7fad0c199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 -- 172.21.15.32:0/2652830664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 msgr2=0x7fad0c1a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 0x7fad0c1a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 -- 172.21.15.32:0/2652830664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c1a4e00 con 0x7fad0c1010a0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad110bb640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 0x7fad0c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad120bd640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 0x7fad0c1a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad118bc640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fad0800d950 tx=0x7fad0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08017030 con 0x7fad0c1010a0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad08004020 con 0x7fad0c1010a0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c1a50f0 con 0x7fad0c1010a0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad080047a0 con 0x7fad0c1010a0 2024-04-23T18:27:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.791+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c1a5610 con 0x7fad0c1010a0 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.792+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c102570 con 0x7fad0c1010a0 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.793+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fad08009a30 con 0x7fad0c1010a0 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.794+0000 7facfeffd640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 0x7facf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.794+0000 7fad110bb640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 0x7facf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.794+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7fad080d0350 con 0x7fad0c1010a0 2024-04-23T18:27:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.794+0000 7fad110bb640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 0x7facf80a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad00006010 tx=0x7fad00000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:02.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:02.799+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad0809c9b0 con 0x7fad0c1010a0 2024-04-23T18:27:03.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:02 smithi032 ceph-mon[29221]: pgmap v371: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 531 KiB/s wr, 71 op/s 2024-04-23T18:27:03.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.023+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad0c101a50 con 0x7fad0c1010a0 2024-04-23T18:27:03.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.024+0000 7facfeffd640 1 -- 172.21.15.32:0/2652830664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11855 (secure 0 0 0) 0x7fad08096e20 con 0x7fad0c1010a0 2024-04-23T18:27:03.017 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:27:03.017 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:26:05.875467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:27:03.018 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-04-23T18:27:03.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 msgr2=0x7facf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 0x7facf80a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad00006010 tx=0x7fad00000bc0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 msgr2=0x7fad0c199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fad0800d950 tx=0x7fad0800de10 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 shutdown_connections 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c19a050 0x7fad0c1a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad0c106170 0x7fad0c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7facf80a3d20 0x7facf80a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 --2- 172.21.15.32:0/2652830664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad0c1010a0 0x7fad0c199580 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 >> 172.21.15.32:0/2652830664 conn(0x7fad0c077f50 msgr2=0x7fad0c0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 shutdown_connections 2024-04-23T18:27:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.026+0000 7fad13b20640 1 -- 172.21.15.32:0/2652830664 wait complete. 2024-04-23T18:27:03.030 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 198bd97f --release --await' 2024-04-23T18:27:03.031 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 198bd97f --release --await 2024-04-23T18:27:03.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:02 smithi040 ceph-mon[32626]: pgmap v371: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 531 KiB/s wr, 71 op/s 2024-04-23T18:27:03.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 -- 172.21.15.32:0/1069134735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54102b30 msgr2=0x7f2e54106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1069134735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54102b30 0x7f2e54106b70 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009900 tx=0x7f2e4802a410 comp rx=0 tx=0).stop 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 -- 172.21.15.32:0/1069134735 shutdown_connections 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1069134735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 0x7f2e5410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1069134735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54102b30 0x7f2e54106b70 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1069134735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 0x7f2e54102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 -- 172.21.15.32:0/1069134735 >> 172.21.15.32:0/1069134735 conn(0x7f2e540fda00 msgr2=0x7f2e540ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 -- 172.21.15.32:0/1069134735 shutdown_connections 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.147+0000 7f2e598a7640 1 -- 172.21.15.32:0/1069134735 wait complete. 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 Processor -- start 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 -- start start 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 0x7f2e54074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 0x7f2e54074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541121e0 con 0x7f2e54074cd0 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54112060 con 0x7f2e54107250 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e598a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54112360 con 0x7f2e54102180 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.148+0000 7f2e537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49834/0 (socket says 172.21.15.32:49834) 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 -- 172.21.15.32:0/1846125674 learned_addr learned my addr 172.21.15.32:0/1846125674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 0x7f2e54074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e527fc640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 0x7f2e54074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 -- 172.21.15.32:0/1846125674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 msgr2=0x7f2e54074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 0x7f2e54074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 -- 172.21.15.32:0/1846125674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 msgr2=0x7f2e54074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 0x7f2e54074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 -- 172.21.15.32:0/1846125674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541a7090 con 0x7f2e54074cd0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e537fe640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009fd0 tx=0x7f2e4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e588a5640 1 -- 172.21.15.32:0/1846125674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c017070 con 0x7f2e54074cd0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e588a5640 1 -- 172.21.15.32:0/1846125674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e4c003e20 con 0x7f2e54074cd0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541a7380 con 0x7f2e54074cd0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e588a5640 1 -- 172.21.15.32:0/1846125674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c010460 con 0x7f2e54074cd0 2024-04-23T18:27:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.149+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e541a7850 con 0x7f2e54074cd0 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e588a5640 1 -- 172.21.15.32:0/1846125674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2e4c003970 con 0x7f2e54074cd0 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 Processor -- start 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 -- start start 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b34f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b37870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e790a8f6d0 con 0x55e790b2a490 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e790a2c1a0 con 0x55e790b2adb0 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e790a8b690 con 0x55e790b35480 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49848/0 (socket says 172.21.15.32:49848) 2024-04-23T18:27:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e527fc640 1 -- 172.21.15.32:0/399525453 learned_addr learned my addr 172.21.15.32:0/399525453 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.152+0000 7f2e588a5640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 0x7f2e3c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 -- 172.21.15.32:0/399525453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 msgr2=0x55e790b37870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b37870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e537fe640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 0x7f2e3c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 -- 172.21.15.32:0/399525453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 msgr2=0x55e790b34f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b34f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b34f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 -- 172.21.15.32:0/399525453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e790a365b0 con 0x55e790b2a490 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e588a5640 1 -- 172.21.15.32:0/1846125674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f2e4c0d04f0 con 0x7f2e54074cd0 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e527fc640 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2e48031440 tx=0x7f2e48005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e796c6451b3700 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e537fe640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 0x7f2e3c0a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0988b0 tx=0x7f2e4c004480 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e39ffb640 1 -- 172.21.15.32:0/399525453 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48041020 con 0x55e790b2a490 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e39ffb640 1 -- 172.21.15.32:0/399525453 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e480044d0 con 0x55e790b2a490 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.153+0000 7f2e39ffb640 1 -- 172.21.15.32:0/399525453 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48035070 con 0x55e790b2a490 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/399525453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 msgr2=0x55e790b2a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2e48031440 tx=0x7f2e48005db0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/399525453 shutdown_connections 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b37870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b34f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/399525453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b2a870 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/399525453 >> 172.21.15.32:0/399525453 conn(0x55e790b267b0 msgr2=0x55e790b26b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/399525453 shutdown_connections 2024-04-23T18:27:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/399525453 wait complete. 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 -- start start 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.154+0000 7f2e5c5c1740 1 Processor -- start 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b34f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b32200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e790b382b0 con 0x55e790b35480 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e790a29b20 con 0x55e790b2a490 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e790b38430 con 0x55e790b2adb0 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 10 client.0 _ll_register_callbacks cb 0x55e790a29530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49864/0 (socket says 172.21.15.32:49864) 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 -- 172.21.15.32:0/2873093073 learned_addr learned my addr 172.21.15.32:0/2873093073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b32200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b34f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 -- 172.21.15.32:0/2873093073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 msgr2=0x55e790b32200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b32200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 -- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 msgr2=0x55e790b34f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b34f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 -- 172.21.15.32:0/2873093073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7907a1920 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e527fc640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2e48004d30 tx=0x7f2e4802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834994d31a6f5034 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48035070 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f2e480316b0 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.155+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e480044d0 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2e48041070 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 10 client.16956 fetch_fsmap learned FSMap version 42 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e79086dd30 con 0x55e790b35480 2024-04-23T18:27:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e480044d0 con 0x55e790b35480 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f2e4803d930 con 0x55e790b35480 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 10 client.16956 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 10 client.16956 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 20 client.16956 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790b454f0 0x55e790b478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 4 client.16956 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e790a4d500 con 0x55e790b454f0 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.156+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790b454f0 0x55e790b478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.157+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790b454f0 0x55e790b478e0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3ff409445d7c050 server_cookie=50935bf232a6c333 in_seq=0 out_seq=0 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.157+0000 7f2e32ffd640 10 client.16956 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e32ffd640 1 -- 172.21.15.32:0/2873093073 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e790a4d500 con 0x55e790b454f0 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e32ffd640 10 client.16956 handle_command_reply: tid=0 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 client.16956 shutdown 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 msgr2=0x55e790b328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2e48004d30 tx=0x7f2e4802a410 comp rx=0 tx=0).stop 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 shutdown_connections 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790b454f0 0x55e790b478e0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b328f0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b32200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e32ffd640 0 client.16956 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/2873093073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b34f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 >> 172.21.15.32:0/2873093073 conn(0x55e790b267b0 msgr2=0x55e790b2c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 shutdown_connections 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/2873093073 wait complete. 2024-04-23T18:27:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.158+0000 7f2e5c5c1740 20 client.16956 trim_cache size 0 max 16384 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 Processor -- start 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 -- start start 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b2c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b2c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541b25f0 con 0x55e790b2a490 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541afe30 con 0x55e790b2adb0 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.175+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541adca0 con 0x55e790b35480 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49874/0 (socket says 172.21.15.32:49874) 2024-04-23T18:27:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 -- 172.21.15.32:0/3980719911 learned_addr learned my addr 172.21.15.32:0/3980719911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 -- 172.21.15.32:0/3980719911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 msgr2=0x55e790b2c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b2c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 -- 172.21.15.32:0/3980719911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 msgr2=0x55e790b2c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b2c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 -- 172.21.15.32:0/3980719911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e790b2abe0 con 0x55e790b2a490 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e527fc640 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2e48055040 tx=0x7f2e48000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0110fbf00028c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e397fa640 1 -- 172.21.15.32:0/3980719911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48035070 con 0x55e790b2a490 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e397fa640 1 -- 172.21.15.32:0/3980719911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e480044d0 con 0x55e790b2a490 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.176+0000 7f2e397fa640 1 -- 172.21.15.32:0/3980719911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e480316b0 con 0x55e790b2a490 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/3980719911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 msgr2=0x55e790b26150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2e48055040 tx=0x7f2e48000bc0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/3980719911 shutdown_connections 2024-04-23T18:27:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b35480 0x55e790b2c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2adb0 0x55e790b2c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/3980719911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b2a490 0x55e790b26150 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/3980719911 >> 172.21.15.32:0/3980719911 conn(0x55e790b267b0 msgr2=0x55e790b2bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/3980719911 shutdown_connections 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/3980719911 wait complete. 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- start start 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 Processor -- start 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b4ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b46ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541ae2f0 con 0x55e790b35480 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541af760 con 0x55e790b2a490 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.177+0000 7f2e5c5c1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2e541ae9c0 con 0x55e790b2adb0 2024-04-23T18:27:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e5c5c1740 10 client.0 _ll_register_callbacks cb 0x7f2e48005520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49878/0 (socket says 172.21.15.32:49878) 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 -- 172.21.15.32:0/1829817403 learned_addr learned my addr 172.21.15.32:0/1829817403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b4ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b46ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 -- 172.21.15.32:0/1829817403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 msgr2=0x55e790b46ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b46ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 -- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 msgr2=0x55e790b4ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b4ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 -- 172.21.15.32:0/1829817403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e790b47d00 con 0x55e790b35480 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e527fc640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2e48003710 tx=0x7f2e480056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df06b67ae7a008 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48035040 con 0x55e790b35480 2024-04-23T18:27:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f2e480318b0 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e480044d0 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2e48031ab0 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e5c5c1740 10 client.16962 fetch_fsmap learned FSMap version 42 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.178+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e790bac6b0 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e480044d0 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f2e4805a040 con 0x55e790b35480 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 10 client.16962 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 10 client.16962 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 20 client.16962 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790bacc70 0x55e790baf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 4 client.16962 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "198bd97f", "release": true, "await": true}] 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "198bd97f", "release": true, "await": true}) -- 0x55e790a9d0f0 con 0x55e790bacc70 2024-04-23T18:27:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790bacc70 0x55e790baf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e52ffd640 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790bacc70 0x55e790baf090 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ed169a931d572b server_cookie=86ff064305ee6e62 in_seq=0 out_seq=0 2024-04-23T18:27:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.179+0000 7f2e32ffd640 10 client.16962 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.178 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:02 smithi064 ceph-mon[33062]: pgmap v371: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 531 KiB/s wr, 71 op/s 2024-04-23T18:27:03.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e32ffd640 1 -- 172.21.15.32:0/1829817403 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e790a9d0f0 con 0x55e790bacc70 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e32ffd640 10 client.16962 handle_command_reply: tid=0 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 client.16962 shutdown 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 msgr2=0x55e790b47530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2e48003710 tx=0x7f2e480056e0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 shutdown_connections 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e790bacc70 0x55e790baf090 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e790b35480 0x55e790b47530 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e790b2adb0 0x55e790b46ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e32ffd640 0 client.16962 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 --2- 172.21.15.32:0/1829817403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e790b2a490 0x55e790b4ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 >> 172.21.15.32:0/1829817403 conn(0x55e790b267b0 msgr2=0x55e790b2c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 shutdown_connections 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.191+0000 7f2e5c5c1740 1 -- 172.21.15.32:0/1829817403 wait complete. 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e5c5c1740 20 client.16962 trim_cache size 0 max 16384 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 42, 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 44, 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "198bd97f": { 2024-04-23T18:27:03.186 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 44, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.0, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:27:03.187 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:27:03.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:27:03.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:27:03.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:27:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 msgr2=0x7f2e3c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 0x7f2e3c0a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0988b0 tx=0x7f2e4c004480 comp rx=0 tx=0).stop 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 msgr2=0x7f2e541a6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009fd0 tx=0x7f2e4c00b260 comp rx=0 tx=0).stop 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 shutdown_connections 2024-04-23T18:27:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2e3c0a3d50 0x7f2e3c0a6210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e54074cd0 0x7f2e541a6990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e54107250 0x7f2e54074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 --2- 172.21.15.32:0/1846125674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e54102180 0x7f2e54074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 >> 172.21.15.32:0/1846125674 conn(0x7f2e540fda00 msgr2=0x7f2e540fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 shutdown_connections 2024-04-23T18:27:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:27:03.192+0000 7f2e598a7640 1 -- 172.21.15.32:0/1846125674 wait complete. 2024-04-23T18:27:03.198 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '198bd97f', seconds elapsed: 31.5 2024-04-23T18:27:03.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 -- 172.21.15.32:0/3319258790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 msgr2=0x7f2790103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.355 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 --2- 172.21.15.32:0/3319258790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f2790103570 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2780009900 tx=0x7f278002a480 comp rx=0 tx=0).stop 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 -- 172.21.15.32:0/3319258790 shutdown_connections 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 --2- 172.21.15.32:0/3319258790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f279010c5b0 0x7f279010ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 --2- 172.21.15.32:0/3319258790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2790103ab0 0x7f279010bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 --2- 172.21.15.32:0/3319258790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f2790103570 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 -- 172.21.15.32:0/3319258790 >> 172.21.15.32:0/3319258790 conn(0x7f27900fc970 msgr2=0x7f27900fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 -- 172.21.15.32:0/3319258790 shutdown_connections 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.360+0000 7f2795929640 1 -- 172.21.15.32:0/3319258790 wait complete. 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 Processor -- start 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 -- start start 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2790103ab0 0x7f27901a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.356 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f279010c5b0 0x7f279019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2790114470 con 0x7f2790103190 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27901142f0 con 0x7f279010c5b0 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f2795929640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f27901145f0 con 0x7f2790103ab0 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49892/0 (socket says 172.21.15.32:49892) 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 -- 172.21.15.32:0/1745584195 learned_addr learned my addr 172.21.15.32:0/1745584195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2790103ab0 0x7f27901a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278f7fe640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f279010c5b0 0x7f279019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 -- 172.21.15.32:0/1745584195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2790103ab0 msgr2=0x7f27901a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2790103ab0 0x7f27901a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 -- 172.21.15.32:0/1745584195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f279010c5b0 msgr2=0x7f279019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f279010c5b0 0x7f279019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.361+0000 7f278effd640 1 -- 172.21.15.32:0/1745584195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279019b000 con 0x7f2790103190 2024-04-23T18:27:03.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f278effd640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f2780031440 tx=0x7f2780003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f2794927640 1 -- 172.21.15.32:0/1745584195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278003f070 con 0x7f2790103190 2024-04-23T18:27:03.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f2794927640 1 -- 172.21.15.32:0/1745584195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2780031b50 con 0x7f2790103190 2024-04-23T18:27:03.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f2794927640 1 -- 172.21.15.32:0/1745584195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780035030 con 0x7f2790103190 2024-04-23T18:27:03.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279019b290 con 0x7f2790103190 2024-04-23T18:27:03.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.362+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27901a7510 con 0x7f2790103190 2024-04-23T18:27:03.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.364+0000 7f2794927640 1 -- 172.21.15.32:0/1745584195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2780042070 con 0x7f2790103190 2024-04-23T18:27:03.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.365+0000 7f2794927640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 0x7f27780a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.366+0000 7f2794927640 1 -- 172.21.15.32:0/1745584195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f27800f0e30 con 0x7f2790103190 2024-04-23T18:27:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.366+0000 7f2798641740 1 Processor -- start 2024-04-23T18:27:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 0x7f27780a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 -- start start 2024-04-23T18:27:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5577b9545970 con 0x5577b95eb800 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5577b94e24a0 con 0x5577b95e0f40 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f2798641740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5577b9541930 con 0x5577b95e0620 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:52328/0 (socket says 172.21.15.32:52328) 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278effd640 1 -- 172.21.15.32:0/103799703 learned_addr learned my addr 172.21.15.32:0/103799703 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278f7fe640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 0x7f27780a6220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f27900ff070 tx=0x7f2784005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278e7fc640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278effd640 1 -- 172.21.15.32:0/103799703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 msgr2=0x5577b95eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.367+0000 7f278effd640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f278effd640 1 -- 172.21.15.32:0/103799703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 msgr2=0x5577b95edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f278effd640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f278effd640 1 -- 172.21.15.32:0/103799703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b9258210 con 0x5577b95e0620 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f278e7fc640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f278effd640 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27800b9250 tx=0x7f278002a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d40349487776b26b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f2775ffb640 1 -- 172.21.15.32:0/103799703 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800d6070 con 0x5577b95e0620 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f2775ffb640 1 -- 172.21.15.32:0/103799703 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27800c0070 con 0x5577b95e0620 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.368+0000 7f2775ffb640 1 -- 172.21.15.32:0/103799703 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780049040 con 0x5577b95e0620 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- 172.21.15.32:0/103799703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 msgr2=0x5577b95e0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27800b9250 tx=0x7f278002a980 comp rx=0 tx=0).stop 2024-04-23T18:27:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- 172.21.15.32:0/103799703 shutdown_connections 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 --2- 172.21.15.32:0/103799703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95e0a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- 172.21.15.32:0/103799703 >> 172.21.15.32:0/103799703 conn(0x5577b95dcb30 msgr2=0x5577b95dcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- 172.21.15.32:0/103799703 shutdown_connections 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- 172.21.15.32:0/103799703 wait complete. 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.369+0000 7f2798641740 1 -- start start 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 Processor -- start 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 0x5577b95e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5577b95ee1f0 con 0x5577b95eb800 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5577b94e00b0 con 0x5577b95e0620 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5577b95ee370 con 0x5577b95e0f40 2024-04-23T18:27:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f2798641740 10 client.0 _ll_register_callbacks cb 0x5577b94cdd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:52422/0 (socket says 172.21.15.32:52422) 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278effd640 1 -- 172.21.15.32:0/3334553193 learned_addr learned my addr 172.21.15.32:0/3334553193 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278e7fc640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 0x5577b95e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 -- 172.21.15.32:0/3334553193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 msgr2=0x5577b95e84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 0x5577b95e84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 -- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 msgr2=0x5577b95eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278f7fe640 1 -- 172.21.15.32:0/3334553193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b9324620 con 0x5577b95eb800 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278e7fc640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 0x5577b95e84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.370+0000 7f278effd640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f278f7fe640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95e8af0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x5577b95ddcc0 tx=0x7f2788012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cbe13f8d33f768 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f276effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2788003970 con 0x5577b95eb800 2024-04-23T18:27:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f2788003b10 con 0x5577b95eb800 2024-04-23T18:27:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27800c0070 con 0x5577b95eb800 2024-04-23T18:27:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2788003d10 con 0x5577b95eb800 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f2798641740 10 client.16974 fetch_fsmap learned FSMap version 42 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5577b95fa1b0 con 0x5577b95eb800 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.371+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27800c0070 con 0x5577b95eb800 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f27880045c0 con 0x5577b95eb800 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 10 client.16974 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 10 client.16974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 20 client.16974 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b95fb6e0 0x5577b95fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 4 client.16974 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577b9553390 con 0x5577b95fb6e0 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.372+0000 7f278effd640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b95fb6e0 0x5577b95fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.373+0000 7f276effd640 10 client.16974 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.373+0000 7f278effd640 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b95fb6e0 0x5577b95fdb60 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f3a561b40ee0cab server_cookie=58ca2b5b9ea1fe45 in_seq=0 out_seq=0 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.373+0000 7f276effd640 1 -- 172.21.15.32:0/3334553193 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5577b9553390 con 0x5577b95fb6e0 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.373+0000 7f276effd640 10 client.16974 handle_command_reply: tid=0 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 client.16974 shutdown 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 msgr2=0x5577b95e8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95e8af0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x5577b95ddcc0 tx=0x7f2788012e70 comp rx=0 tx=0).stop 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 shutdown_connections 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b95fb6e0 0x5577b95fdb60 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95eb800 0x5577b95e8af0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0f40 0x5577b95e84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f276effd640 0 client.16974 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 --2- 172.21.15.32:0/3334553193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0620 0x5577b95eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 >> 172.21.15.32:0/3334553193 conn(0x5577b95dcb30 msgr2=0x5577b95e2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 shutdown_connections 2024-04-23T18:27:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 1 -- 172.21.15.32:0/3334553193 wait complete. 2024-04-23T18:27:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.374+0000 7f2798641740 20 client.16974 trim_cache size 0 max 16384 2024-04-23T18:27:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 Processor -- start 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 -- start start 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 0x5577b95e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95eb800 0x5577b9603480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f27901b2700 con 0x5577b95e0f40 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27901aff40 con 0x5577b95eb800 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.392+0000 7f2798641740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f27901addb0 con 0x5577b95e0620 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 0x5577b95e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:52348/0 (socket says 172.21.15.32:52348) 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 0x5577b95e7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49904/0 (socket says 172.21.15.32:49904) 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 -- 172.21.15.32:0/3186875538 learned_addr learned my addr 172.21.15.32:0/3186875538 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278f7fe640 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95eb800 0x5577b9603480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 -- 172.21.15.32:0/3186875538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95eb800 msgr2=0x5577b9603480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95eb800 0x5577b9603480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 -- 172.21.15.32:0/3186875538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 msgr2=0x5577b95e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 0x5577b95e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 -- 172.21.15.32:0/3186875538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b95e18b0 con 0x5577b95e0620 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f278effd640 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f27800bcaf0 tx=0x7f27800dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14eef9a4d6d3efd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f27757fa640 1 -- 172.21.15.32:0/3186875538 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800db070 con 0x5577b95e0620 2024-04-23T18:27:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f27757fa640 1 -- 172.21.15.32:0/3186875538 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27800c0040 con 0x5577b95e0620 2024-04-23T18:27:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.393+0000 7f27757fa640 1 -- 172.21.15.32:0/3186875538 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800a6030 con 0x5577b95e0620 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.394+0000 7f2798641740 1 -- 172.21.15.32:0/3186875538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 msgr2=0x5577b95f7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.394+0000 7f2798641740 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f27800bcaf0 tx=0x7f27800dd040 comp rx=0 tx=0).stop 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 -- 172.21.15.32:0/3186875538 shutdown_connections 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95eb800 0x5577b9603480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0f40 0x5577b95e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 --2- 172.21.15.32:0/3186875538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95e0620 0x5577b95f7020 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 -- 172.21.15.32:0/3186875538 >> 172.21.15.32:0/3186875538 conn(0x5577b95dcb30 msgr2=0x5577b95db420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 -- 172.21.15.32:0/3186875538 shutdown_connections 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 -- 172.21.15.32:0/3186875538 wait complete. 2024-04-23T18:27:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.395+0000 7f2798641740 1 -- start start 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 Processor -- start 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 0x5577b95f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f27901ae400 con 0x5577b95e0620 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27901af870 con 0x5577b95e0f40 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f27901aead0 con 0x5577b95eb800 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49910/0 (socket says 172.21.15.32:49910) 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f2798641740 10 client.0 _ll_register_callbacks cb 0x7f2788012620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 -- 172.21.15.32:0/1065669221 learned_addr learned my addr 172.21.15.32:0/1065669221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278f7fe640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 0x5577b95f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 -- 172.21.15.32:0/1065669221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 msgr2=0x5577b95f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 0x5577b95f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 -- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 msgr2=0x5577b95f1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.396+0000 7f278effd640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95f1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f278effd640 1 -- 172.21.15.32:0/1065669221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b9660d80 con 0x5577b95e0620 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f278f7fe640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 0x5577b95f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f278effd640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2780004650 tx=0x7f278002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13143dffdf944139 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f276effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800a0060 con 0x5577b95e0620 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8484+0+0 (secure 0 0 0) 0x7f27800ee820 con 0x5577b95e0620 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27800c0040 con 0x5577b95e0620 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2780073030 con 0x5577b95e0620 2024-04-23T18:27:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f2798641740 10 client.16980 fetch_fsmap learned FSMap version 42 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5577b9662070 con 0x5577b95e0620 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.397+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27800d61e0 con 0x5577b95e0620 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f27800c55d0 con 0x5577b95e0620 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 10 client.16980 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 10 client.16980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 20 client.16980 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b9662550 0x5577b9664970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 4 client.16980 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f27800c0040 con 0x5577b9662550 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b9662550 0x5577b9664970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f278e7fc640 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b9662550 0x5577b9664970 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c74705c4f27ed6d8 server_cookie=5bcaf42263999e3a in_seq=0 out_seq=0 2024-04-23T18:27:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.398+0000 7f276effd640 10 client.16980 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 695.0 s, 123.9 tps, lat 4993.197 ms stddev 11290.784 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.914+0000 7f276effd640 1 -- 172.21.15.32:0/1065669221 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f27800c0040 con 0x5577b9662550 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.914+0000 7f276effd640 10 client.16980 handle_command_reply: tid=0 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.914+0000 7f2798641740 1 client.16980 shutdown 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 msgr2=0x5577b95f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2780004650 tx=0x7f278002a480 comp rx=0 tx=0).stop 2024-04-23T18:27:03.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 shutdown_connections 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5577b9662550 0x5577b9664970 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5577b95eb800 0x5577b95f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b95e0f40 0x5577b95f1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 --2- 172.21.15.32:0/1065669221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577b95e0620 0x5577b95f0fe0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 >> 172.21.15.32:0/1065669221 conn(0x5577b95dcb30 msgr2=0x5577b95e74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f276effd640 0 client.16980 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 shutdown_connections 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 1 -- 172.21.15.32:0/1065669221 wait complete. 2024-04-23T18:27:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.915+0000 7f2798641740 20 client.16980 trim_cache size 0 max 16384 2024-04-23T18:27:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:27:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:27:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:27:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:27:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.916+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 msgr2=0x7f27780a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.916+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 0x7f27780a6220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f27900ff070 tx=0x7f2784005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 msgr2=0x7f27901a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f2780031440 tx=0x7f2780003b90 comp rx=0 tx=0).stop 2024-04-23T18:27:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 shutdown_connections 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f279010c5b0 0x7f279019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2790103ab0 0x7f27901a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f27780a3d60 0x7f27780a6220 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 --2- 172.21.15.32:0/1745584195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2790103190 0x7f27901a06c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 >> 172.21.15.32:0/1745584195 conn(0x7f27900fc970 msgr2=0x7f27900fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 shutdown_connections 2024-04-23T18:27:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:03.917+0000 7f2795929640 1 -- 172.21.15.32:0/1745584195 wait complete. 2024-04-23T18:27:04.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:03 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2652830664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:04.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:03 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2652830664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:04.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:03 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2652830664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:05.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:04 smithi032 ceph-mon[29221]: pgmap v372: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 558 KiB/s wr, 87 op/s 2024-04-23T18:27:05.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:04 smithi040 ceph-mon[32626]: pgmap v372: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 558 KiB/s wr, 87 op/s 2024-04-23T18:27:05.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:04 smithi064 ceph-mon[33062]: pgmap v372: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 558 KiB/s wr, 87 op/s 2024-04-23T18:27:07.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:06 smithi032 ceph-mon[29221]: pgmap v373: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 885 KiB/s wr, 125 op/s 2024-04-23T18:27:07.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:06 smithi040 ceph-mon[32626]: pgmap v373: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 885 KiB/s wr, 125 op/s 2024-04-23T18:27:07.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:06 smithi064 ceph-mon[33062]: pgmap v373: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 885 KiB/s wr, 125 op/s 2024-04-23T18:27:08.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:07 smithi032 ceph-mon[29221]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:27:08.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:07 smithi040 ceph-mon[32626]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:27:08.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:07 smithi064 ceph-mon[33062]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:27:08.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 700.0 s, 2211.0 tps, lat 49.797 ms stddev 307.035 2024-04-23T18:27:09.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:08 smithi032 ceph-mon[29221]: pgmap v375: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 444 KiB/s wr, 70 op/s 2024-04-23T18:27:09.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:08 smithi040 ceph-mon[32626]: pgmap v375: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 444 KiB/s wr, 70 op/s 2024-04-23T18:27:09.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:08 smithi064 ceph-mon[33062]: pgmap v375: 97 pgs: 97 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 444 KiB/s wr, 70 op/s 2024-04-23T18:27:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:10 smithi032 ceph-mon[29221]: pgmap v376: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.9 MiB/s wr, 394 op/s 2024-04-23T18:27:11.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:10 smithi032 ceph-mon[29221]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:27:11.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:10 smithi040 ceph-mon[32626]: pgmap v376: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.9 MiB/s wr, 394 op/s 2024-04-23T18:27:11.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:10 smithi040 ceph-mon[32626]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:27:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:10 smithi064 ceph-mon[33062]: pgmap v376: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.9 MiB/s wr, 394 op/s 2024-04-23T18:27:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:10 smithi064 ceph-mon[33062]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:27:13.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:12 smithi032 ceph-mon[29221]: pgmap v378: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.8 MiB/s wr, 469 op/s 2024-04-23T18:27:13.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:12 smithi040 ceph-mon[32626]: pgmap v378: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.8 MiB/s wr, 469 op/s 2024-04-23T18:27:13.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:12 smithi064 ceph-mon[33062]: pgmap v378: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.8 MiB/s wr, 469 op/s 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 705.0 s, 2564.0 tps, lat 34.870 ms stddev 38.944 2024-04-23T18:27:14.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:13 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:14.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:13 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:14.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:13 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:15.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:14 smithi040 ceph-mon[32626]: pgmap v379: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.4 MiB/s wr, 416 op/s 2024-04-23T18:27:15.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:14 smithi064 ceph-mon[33062]: pgmap v379: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.4 MiB/s wr, 416 op/s 2024-04-23T18:27:15.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:14 smithi032 ceph-mon[29221]: pgmap v379: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.4 MiB/s wr, 416 op/s 2024-04-23T18:27:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 -- 172.21.15.32:0/978166237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290102b70 msgr2=0x7f1290106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 --2- 172.21.15.32:0/978166237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290102b70 0x7f1290106bb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1280009900 tx=0x7f128002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 -- 172.21.15.32:0/978166237 shutdown_connections 2024-04-23T18:27:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 --2- 172.21.15.32:0/978166237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1290107290 0x7f129010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 --2- 172.21.15.32:0/978166237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290102b70 0x7f1290106bb0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 --2- 172.21.15.32:0/978166237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 0x7f12901025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 -- 172.21.15.32:0/978166237 >> 172.21.15.32:0/978166237 conn(0x7f12900fda40 msgr2=0x7f12900ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 -- 172.21.15.32:0/978166237 shutdown_connections 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.018+0000 7f1296f20640 1 -- 172.21.15.32:0/978166237 wait complete. 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 Processor -- start 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 -- start start 2024-04-23T18:27:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 0x7f12901a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f129019a7b0 0x7f129019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12901121d0 con 0x7f1290107290 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1290112050 con 0x7f12901021c0 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1296f20640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1290112350 con 0x7f129019a7b0 2024-04-23T18:27:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33038/0 (socket says 172.21.15.32:33038) 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 -- 172.21.15.32:0/1960830624 learned_addr learned my addr 172.21.15.32:0/1960830624 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 0x7f12901a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f12954bd640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f129019a7b0 0x7f129019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 -- 172.21.15.32:0/1960830624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f129019a7b0 msgr2=0x7f129019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f129019a7b0 0x7f129019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 -- 172.21.15.32:0/1960830624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 msgr2=0x7f12901a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.019+0000 7f128ffff640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 0x7f12901a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f128ffff640 1 -- 172.21.15.32:0/1960830624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129019b340 con 0x7f1290107290 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f128ffff640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1280032fd0 tx=0x7f1280004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f128dffb640 1 -- 172.21.15.32:0/1960830624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128003f030 con 0x7f1290107290 2024-04-23T18:27:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129019b5d0 con 0x7f1290107290 2024-04-23T18:27:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f128dffb640 1 -- 172.21.15.32:0/1960830624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1280004490 con 0x7f1290107290 2024-04-23T18:27:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f128dffb640 1 -- 172.21.15.32:0/1960830624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1280035070 con 0x7f1290107290 2024-04-23T18:27:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.020+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12901a77b0 con 0x7f1290107290 2024-04-23T18:27:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.022+0000 7f128dffb640 1 -- 172.21.15.32:0/1960830624 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1280042030 con 0x7f1290107290 2024-04-23T18:27:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 Processor -- start 2024-04-23T18:27:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 -- start start 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa69d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa69d800 0x55dbaa69fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa5f7970 con 0x55dbaa69d800 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa5944a0 con 0x55dbaa692620 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1299c36740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa5f3930 con 0x55dbaa692f40 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f128dffb640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 0x7f127c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f128ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa69d800 0x55dbaa69fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:36776/0 (socket says 172.21.15.32:36776) 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 -- 172.21.15.32:0/304296875 learned_addr learned my addr 172.21.15.32:0/304296875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 0x7f127c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f1294cbc640 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa69d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f128dffb640 1 -- 172.21.15.32:0/1960830624 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f12800f0d40 con 0x7f1290107290 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 -- 172.21.15.32:0/304296875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 msgr2=0x55dbaa69d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa69d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 -- 172.21.15.32:0/304296875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa69d800 msgr2=0x55dbaa69fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa69d800 0x55dbaa69fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.023+0000 7f12954bd640 1 -- 172.21.15.32:0/304296875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbaa30a210 con 0x55dbaa692620 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f12954bd640 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1278013b50 tx=0x7f12780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f2a11cf8946f7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f12954bd640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 0x7f127c0a61d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1278010ca0 tx=0x7f1278015000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1286ffd640 1 -- 172.21.15.32:0/304296875 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1278021070 con 0x55dbaa692620 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1286ffd640 1 -- 172.21.15.32:0/304296875 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12780035f0 con 0x55dbaa692620 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1286ffd640 1 -- 172.21.15.32:0/304296875 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127801d630 con 0x55dbaa692620 2024-04-23T18:27:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1299c36740 1 -- 172.21.15.32:0/304296875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 msgr2=0x55dbaa692a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1299c36740 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1278013b50 tx=0x7f12780107a0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1299c36740 1 -- 172.21.15.32:0/304296875 shutdown_connections 2024-04-23T18:27:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1299c36740 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa69d800 0x55dbaa69fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.024+0000 7f1299c36740 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa69d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 --2- 172.21.15.32:0/304296875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa692a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- 172.21.15.32:0/304296875 >> 172.21.15.32:0/304296875 conn(0x55dbaa68eb30 msgr2=0x55dbaa68ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- 172.21.15.32:0/304296875 shutdown_connections 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- 172.21.15.32:0/304296875 wait complete. 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- start start 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 Processor -- start 2024-04-23T18:27:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692f40 0x55dbaa69a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa69aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa6a03d0 con 0x55dbaa692620 2024-04-23T18:27:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa5920b0 con 0x55dbaa692f40 2024-04-23T18:27:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dbaa6a0550 con 0x55dbaa69d800 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1299c36740 10 client.0 _ll_register_callbacks cb 0x55dbaa57fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1294cbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1294cbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33064/0 (socket says 172.21.15.32:33064) 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.025+0000 7f1294cbc640 1 -- 172.21.15.32:0/1645200352 learned_addr learned my addr 172.21.15.32:0/1645200352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f128ffff640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692f40 0x55dbaa69a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f12954bd640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa69aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 -- 172.21.15.32:0/1645200352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 msgr2=0x55dbaa69aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa69aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 -- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692f40 msgr2=0x55dbaa69a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692f40 0x55dbaa69a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 -- 172.21.15.32:0/1645200352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbaa3d6620 con 0x55dbaa692620 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f12954bd640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa69aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1288003c20 tx=0x7f1288004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936bdfad50a6d36a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f126ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288011850 con 0x55dbaa692620 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f12880119f0 con 0x55dbaa692620 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12780035f0 con 0x55dbaa692620 2024-04-23T18:27:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1288003e50 con 0x55dbaa692620 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1299c36740 10 client.16992 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55dbaa6ac3e0 con 0x55dbaa692620 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.026+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12780035f0 con 0x55dbaa692620 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f1288022020 con 0x55dbaa692620 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 10 client.16992 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 10 client.16992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 20 client.16992 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa6ad850 0x55dbaa6afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 4 client.16992 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbaa605390 con 0x55dbaa6ad850 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f128ffff640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa6ad850 0x55dbaa6afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f126ffff640 10 client.16992 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.027+0000 7f128ffff640 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa6ad850 0x55dbaa6afc60 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9295e67814466897 server_cookie=6684e826a5388008 in_seq=0 out_seq=0 2024-04-23T18:27:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f126ffff640 1 -- 172.21.15.32:0/1645200352 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbaa605390 con 0x55dbaa6ad850 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f126ffff640 10 client.16992 handle_command_reply: tid=0 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 client.16992 shutdown 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 msgr2=0x55dbaa69d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1288003c20 tx=0x7f1288004410 comp rx=0 tx=0).stop 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 shutdown_connections 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa6ad850 0x55dbaa6afc60 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa69aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f126ffff640 0 client.16992 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692f40 0x55dbaa69a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 --2- 172.21.15.32:0/1645200352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa69d1f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 >> 172.21.15.32:0/1645200352 conn(0x55dbaa68eb30 msgr2=0x55dbaa6948b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 shutdown_connections 2024-04-23T18:27:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 1 -- 172.21.15.32:0/1645200352 wait complete. 2024-04-23T18:27:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.028+0000 7f1299c36740 20 client.16992 trim_cache size 0 max 16384 2024-04-23T18:27:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 Processor -- start 2024-04-23T18:27:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 -- start start 2024-04-23T18:27:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa694730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 0x55dbaa69cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12901b26c0 con 0x55dbaa692620 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12901aff00 con 0x55dbaa69d800 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1299c36740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f12901add70 con 0x55dbaa692f40 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1294cbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1294cbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33068/0 (socket says 172.21.15.32:33068) 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.043+0000 7f1294cbc640 1 -- 172.21.15.32:0/1533866493 learned_addr learned my addr 172.21.15.32:0/1533866493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f12954bd640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 0x55dbaa69cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f128ffff640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa694730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 -- 172.21.15.32:0/1533866493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 msgr2=0x55dbaa694730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa694730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 -- 172.21.15.32:0/1533866493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 msgr2=0x55dbaa69cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 0x55dbaa69cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 -- 172.21.15.32:0/1533866493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbaa699490 con 0x55dbaa692620 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f12954bd640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 0x55dbaa69cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f1294cbc640 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1288003bf0 tx=0x7f12880059c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8330fa3afc36d82b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f12867fc640 1 -- 172.21.15.32:0/1533866493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288011650 con 0x55dbaa692620 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f12867fc640 1 -- 172.21.15.32:0/1533866493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1288011ca0 con 0x55dbaa692620 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.044+0000 7f12867fc640 1 -- 172.21.15.32:0/1533866493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288024a00 con 0x55dbaa692620 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- 172.21.15.32:0/1533866493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 msgr2=0x55dbaa68e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1288003bf0 tx=0x7f12880059c0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- 172.21.15.32:0/1533866493 shutdown_connections 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa69d800 0x55dbaa69cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa692f40 0x55dbaa694730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- 172.21.15.32:0/1533866493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692620 0x55dbaa68e430 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- 172.21.15.32:0/1533866493 >> 172.21.15.32:0/1533866493 conn(0x55dbaa68eb30 msgr2=0x55dbaa694020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- 172.21.15.32:0/1533866493 shutdown_connections 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- 172.21.15.32:0/1533866493 wait complete. 2024-04-23T18:27:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- start start 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 Processor -- start 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa6a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa6a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12901ae3c0 con 0x55dbaa692f40 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12901af830 con 0x55dbaa692620 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.045+0000 7f1299c36740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f12901aea90 con 0x55dbaa69d800 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1299c36740 10 client.0 _ll_register_callbacks cb 0x7f1288009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33082/0 (socket says 172.21.15.32:33082) 2024-04-23T18:27:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 -- 172.21.15.32:0/2440013956 learned_addr learned my addr 172.21.15.32:0/2440013956 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1294cbc640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa6a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f12954bd640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa6a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 -- 172.21.15.32:0/2440013956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 msgr2=0x55dbaa6a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa6a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 -- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 msgr2=0x55dbaa6a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa6a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 -- 172.21.15.32:0/2440013956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbaa6a1e30 con 0x55dbaa692f40 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1294cbc640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa6a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f128ffff640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12800b9100 tx=0x7f1280031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745edf76afc83d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f126ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1280049040 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f128003b460 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1288011ca0 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f12800dbe60 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1299c36740 10 client.16998 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.046+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55dbaa713f80 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1288011ca0 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f12800b9310 con 0x55dbaa692f40 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 10 client.16998 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 10 client.16998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 20 client.16998 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa7147e0 0x55dbaa716c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 4 client.16998 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbaa5ff200 con 0x55dbaa7147e0 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1294cbc640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa7147e0 0x55dbaa716c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f126ffff640 10 client.16998 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.047+0000 7f1294cbc640 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa7147e0 0x55dbaa716c00 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c38d7713ead6bf36 server_cookie=556a2151ffbbf5ea in_seq=0 out_seq=0 2024-04-23T18:27:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f126ffff640 1 -- 172.21.15.32:0/2440013956 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbaa5ff200 con 0x55dbaa7147e0 2024-04-23T18:27:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f126ffff640 10 client.16998 handle_command_reply: tid=0 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 client.16998 shutdown 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 msgr2=0x55dbaa6a0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12800b9100 tx=0x7f1280031a10 comp rx=0 tx=0).stop 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 shutdown_connections 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55dbaa7147e0 0x55dbaa716c00 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dbaa69d800 0x55dbaa6a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dbaa692f40 0x55dbaa6a0ff0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 --2- 172.21.15.32:0/2440013956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbaa692620 0x55dbaa6a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f126ffff640 0 client.16998 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 >> 172.21.15.32:0/2440013956 conn(0x55dbaa68eb30 msgr2=0x55dbaa6a8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 shutdown_connections 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 1 -- 172.21.15.32:0/2440013956 wait complete. 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.057+0000 7f1299c36740 20 client.16998 trim_cache size 0 max 16384 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:27:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:27:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 msgr2=0x7f127c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 0x7f127c0a61d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1278010ca0 tx=0x7f1278015000 comp rx=0 tx=0).stop 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 msgr2=0x7f12901a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1280032fd0 tx=0x7f1280004270 comp rx=0 tx=0).stop 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 shutdown_connections 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f129019a7b0 0x7f129019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f127c0a3d10 0x7f127c0a61d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1290107290 0x7f12901a0b80 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 --2- 172.21.15.32:0/1960830624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12901021c0 0x7f12901a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 >> 172.21.15.32:0/1960830624 conn(0x7f12900fda40 msgr2=0x7f12900fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 shutdown_connections 2024-04-23T18:27:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:17.058+0000 7f1296f20640 1 -- 172.21.15.32:0/1960830624 wait complete. 2024-04-23T18:27:17.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:16 smithi040 ceph-mon[32626]: pgmap v380: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 8.1 MiB/s wr, 763 op/s 2024-04-23T18:27:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:16 smithi064 ceph-mon[33062]: pgmap v380: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 8.1 MiB/s wr, 763 op/s 2024-04-23T18:27:17.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:16 smithi032 ceph-mon[29221]: pgmap v380: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 8.1 MiB/s wr, 763 op/s 2024-04-23T18:27:18.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:17 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:18.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:17 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:18.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:17 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 710.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:27:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:18 smithi040 ceph-mon[32626]: pgmap v381: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-23T18:27:19.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:18 smithi064 ceph-mon[33062]: pgmap v381: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-23T18:27:19.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:18 smithi032 ceph-mon[29221]: pgmap v381: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-23T18:27:21.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:20 smithi040 ceph-mon[32626]: pgmap v382: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T18:27:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:20 smithi064 ceph-mon[33062]: pgmap v382: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T18:27:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:20 smithi032 ceph-mon[29221]: pgmap v382: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T18:27:23.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:22 smithi040 ceph-mon[32626]: pgmap v383: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.2 MiB/s wr, 418 op/s 2024-04-23T18:27:23.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:22 smithi064 ceph-mon[33062]: pgmap v383: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.2 MiB/s wr, 418 op/s 2024-04-23T18:27:23.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:22 smithi032 ceph-mon[29221]: pgmap v383: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.2 MiB/s wr, 418 op/s 2024-04-23T18:27:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 715.0 s, 1259.1 tps, lat 167.370 ms stddev 1070.107 2024-04-23T18:27:25.118 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:24 smithi040 ceph-mon[32626]: pgmap v384: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 397 op/s 2024-04-23T18:27:25.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:24 smithi064 ceph-mon[33062]: pgmap v384: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 397 op/s 2024-04-23T18:27:25.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:24 smithi032 ceph-mon[29221]: pgmap v384: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 397 op/s 2024-04-23T18:27:27.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:26 smithi040 ceph-mon[32626]: pgmap v385: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.2 MiB/s wr, 511 op/s 2024-04-23T18:27:27.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:26 smithi064 ceph-mon[33062]: pgmap v385: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.2 MiB/s wr, 511 op/s 2024-04-23T18:27:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:26 smithi032 ceph-mon[29221]: pgmap v385: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.2 MiB/s wr, 511 op/s 2024-04-23T18:27:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 720.0 s, 2955.8 tps, lat 33.443 ms stddev 36.537 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: pgmap v386: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:28 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: pgmap v386: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-04-23T18:27:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:28 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: pgmap v386: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-04-23T18:27:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:29.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:28 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:30.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:29 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:30.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:29 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:30.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:29 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:30.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:29 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:30.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:29 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:30.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:29 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- 172.21.15.32:0/2776048144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 msgr2=0x7f319810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 --2- 172.21.15.32:0/2776048144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319810e9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3190013f10 tx=0x7f3190032a20 comp rx=0 tx=0).stop 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- 172.21.15.32:0/2776048144 shutdown_connections 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 --2- 172.21.15.32:0/2776048144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319810e9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 --2- 172.21.15.32:0/2776048144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3198103ae0 0x7f319810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 --2- 172.21.15.32:0/2776048144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31981031c0 0x7f31981035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- 172.21.15.32:0/2776048144 >> 172.21.15.32:0/2776048144 conn(0x7f3198077f70 msgr2=0x7f3198078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- 172.21.15.32:0/2776048144 shutdown_connections 2024-04-23T18:27:30.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- 172.21.15.32:0/2776048144 wait complete. 2024-04-23T18:27:30.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 Processor -- start 2024-04-23T18:27:30.177 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.183+0000 7f31a04e8640 1 -- start start 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31981031c0 0x7f3198196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3198103ae0 0x7f3198196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3198110020 con 0x7f31981031c0 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f319810fea0 con 0x7f3198103ae0 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f31a04e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31981101a0 con 0x7f319810c590 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319e284640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31981031c0 0x7f3198196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:54794/0 (socket says 172.21.15.32:54794) 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 -- 172.21.15.32:0/4266634993 learned_addr learned my addr 172.21.15.32:0/4266634993 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319da83640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3198103ae0 0x7f3198196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 -- 172.21.15.32:0/4266634993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3198103ae0 msgr2=0x7f3198196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3198103ae0 0x7f3198196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 -- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31981031c0 msgr2=0x7f3198196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31981031c0 0x7f3198196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.184+0000 7f319ea85640 1 -- 172.21.15.32:0/4266634993 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319819b4f0 con 0x7f319810c590 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f319ea85640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3190005ab0 tx=0x7f3190013660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f318f7fe640 1 -- 172.21.15.32:0/4266634993 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319004d070 con 0x7f319810c590 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f318f7fe640 1 -- 172.21.15.32:0/4266634993 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3190048920 con 0x7f319810c590 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f318f7fe640 1 -- 172.21.15.32:0/4266634993 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319003a050 con 0x7f319810c590 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319819b780 con 0x7f319810c590 2024-04-23T18:27:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.185+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31981a3070 con 0x7f319810c590 2024-04-23T18:27:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.189+0000 7f31a3202740 1 Processor -- start 2024-04-23T18:27:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f318f7fe640 1 -- 172.21.15.32:0/4266634993 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3190050070 con 0x7f319810c590 2024-04-23T18:27:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 -- start start 2024-04-23T18:27:30.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85f40 0x5639fba902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba90800 0x5639fba92bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5639fb9ea970 con 0x5639fba85620 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5639fb9874a0 con 0x5639fba90800 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f31a3202740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5639fb9e6930 con 0x5639fba85f40 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.190+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48908/0 (socket says 172.21.15.32:48908) 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 -- 172.21.15.32:0/2074407909 learned_addr learned my addr 172.21.15.32:0/2074407909 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba90800 0x5639fba92bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319e284640 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85f40 0x5639fba902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f318f7fe640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 0x7f31840a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 -- 172.21.15.32:0/2074407909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85f40 msgr2=0x5639fba902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85f40 0x5639fba902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 -- 172.21.15.32:0/2074407909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba90800 msgr2=0x5639fba92bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba90800 0x5639fba92bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 -- 172.21.15.32:0/2074407909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639fb6fd210 con 0x5639fba85620 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f318f7fe640 1 -- 172.21.15.32:0/4266634993 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f31900f8b30 con 0x7f319810c590 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319ea85640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 0x7f31840a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319da83640 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f31880135e0 tx=0x7f318800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472df61d183b4afe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f318cbfe640 1 -- 172.21.15.32:0/2074407909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188019030 con 0x5639fba85620 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.191+0000 7f319ea85640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 0x7f31840a62d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31900cd1d0 tx=0x7f3190013a60 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f318cbfe640 1 -- 172.21.15.32:0/2074407909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318800db40 con 0x5639fba85620 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f318cbfe640 1 -- 172.21.15.32:0/2074407909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318800f9f0 con 0x5639fba85620 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 -- 172.21.15.32:0/2074407909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 msgr2=0x5639fba85a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f31880135e0 tx=0x7f318800a710 comp rx=0 tx=0).stop 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 -- 172.21.15.32:0/2074407909 shutdown_connections 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba90800 0x5639fba92bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85f40 0x5639fba902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 --2- 172.21.15.32:0/2074407909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fba85a00 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 -- 172.21.15.32:0/2074407909 >> 172.21.15.32:0/2074407909 conn(0x5639fba81b30 msgr2=0x5639fba81f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 -- 172.21.15.32:0/2074407909 shutdown_connections 2024-04-23T18:27:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.192+0000 7f31a3202740 1 -- 172.21.15.32:0/2074407909 wait complete. 2024-04-23T18:27:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 -- start start 2024-04-23T18:27:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 Processor -- start 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba8d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fba8d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5639fba93430 con 0x5639fba90800 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5639fb9850b0 con 0x5639fba85f40 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5639fba935b0 con 0x5639fba85620 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f319da83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f319da83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:54818/0 (socket says 172.21.15.32:54818) 2024-04-23T18:27:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f319e284640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba8d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f319da83640 1 -- 172.21.15.32:0/2479555306 learned_addr learned my addr 172.21.15.32:0/2479555306 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f319da83640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fba8d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.193+0000 7f31a3202740 10 client.0 _ll_register_callbacks cb 0x5639fb972d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 -- 172.21.15.32:0/2479555306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 msgr2=0x5639fba90130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 -- 172.21.15.32:0/2479555306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 msgr2=0x5639fba8d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fba8d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 -- 172.21.15.32:0/2479555306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639fb7c9620 con 0x5639fba85f40 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319da83640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f319e284640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba8d420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f319400ab30 tx=0x7f31940111b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b5b4d8ba12bc8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f3181ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319401a070 con 0x5639fba85f40 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318800db40 con 0x5639fba85f40 2024-04-23T18:27:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f3194011d90 con 0x5639fba85f40 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3194026050 con 0x5639fba85f40 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f31a3202740 10 client.26507 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5639fba9f3d0 con 0x5639fba85f40 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.194+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318800db40 con 0x5639fba85f40 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f319401e4d0 con 0x5639fba85f40 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 10 client.26507 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 10 client.26507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 20 client.26507 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbaa0900 0x5639fbaa2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 4 client.26507 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639fb9f8390 con 0x5639fbaa0900 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.195+0000 7f319da83640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbaa0900 0x5639fbaa2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.196+0000 7f3181ffb640 10 client.26507 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.197+0000 7f319da83640 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbaa0900 0x5639fbaa2da0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d9987a6bed284f server_cookie=88f0da4255513516 in_seq=0 out_seq=0 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.197+0000 7f3181ffb640 1 -- 172.21.15.32:0/2479555306 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639fb9f8390 con 0x5639fbaa0900 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.197+0000 7f3181ffb640 10 client.26507 handle_command_reply: tid=0 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.197+0000 7f31a3202740 1 client.26507 shutdown 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 msgr2=0x5639fba8d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba8d420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f319400ab30 tx=0x7f31940111b0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 shutdown_connections 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbaa0900 0x5639fbaa2da0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fba8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba8d420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f3181ffb640 0 client.26507 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 --2- 172.21.15.32:0/2479555306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba90130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 >> 172.21.15.32:0/2479555306 conn(0x5639fba81b30 msgr2=0x5639fba878b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 shutdown_connections 2024-04-23T18:27:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 1 -- 172.21.15.32:0/2479555306 wait complete. 2024-04-23T18:27:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.198+0000 7f31a3202740 20 client.26507 trim_cache size 0 max 16384 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 Processor -- start 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 -- start start 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba9c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba80b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fbaa86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f31981ae1d0 con 0x5639fba90800 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f31981aba10 con 0x5639fba85f40 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f31a3202740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31981a9880 con 0x5639fba85620 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f319da83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba9c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fbaa86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fbaa86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48924/0 (socket says 172.21.15.32:48924) 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f319da83640 1 -- 172.21.15.32:0/718292456 learned_addr learned my addr 172.21.15.32:0/718292456 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.219+0000 7f319e284640 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba80b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.221+0000 7f319da83640 1 -- 172.21.15.32:0/718292456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 msgr2=0x5639fba80b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.221+0000 7f319da83640 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba80b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.221+0000 7f319da83640 1 -- 172.21.15.32:0/718292456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 msgr2=0x5639fbaa86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.221+0000 7f319da83640 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fbaa86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.221+0000 7f319da83640 1 -- 172.21.15.32:0/718292456 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639fba8df40 con 0x5639fba85620 2024-04-23T18:27:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.222+0000 7f319da83640 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba9c2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f318800af90 tx=0x7f318800a2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=385ee597daebe30f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.222+0000 7f3183fff640 1 -- 172.21.15.32:0/718292456 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188034070 con 0x5639fba85620 2024-04-23T18:27:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.222+0000 7f3183fff640 1 -- 172.21.15.32:0/718292456 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318801a040 con 0x5639fba85620 2024-04-23T18:27:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.222+0000 7f3183fff640 1 -- 172.21.15.32:0/718292456 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188019040 con 0x5639fba85620 2024-04-23T18:27:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 -- 172.21.15.32:0/718292456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 msgr2=0x5639fba9c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba9c2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f318800af90 tx=0x7f318800a2d0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 -- 172.21.15.32:0/718292456 shutdown_connections 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba90800 0x5639fbaa86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fba80b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 --2- 172.21.15.32:0/718292456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba85620 0x5639fba9c2f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 -- 172.21.15.32:0/718292456 >> 172.21.15.32:0/718292456 conn(0x5639fba81b30 msgr2=0x5639fba80420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 -- 172.21.15.32:0/718292456 shutdown_connections 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.223+0000 7f31a3202740 1 -- 172.21.15.32:0/718292456 wait complete. 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 -- start start 2024-04-23T18:27:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 Processor -- start 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fbaff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba90800 0x5639fba95af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f31981a9ed0 con 0x5639fba85620 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f31981ab340 con 0x5639fba85f40 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31981aa5a0 con 0x5639fba90800 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f31a3202740 10 client.0 _ll_register_callbacks cb 0x7f319400a770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f319da83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48936/0 (socket says 172.21.15.32:48936) 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f319da83640 1 -- 172.21.15.32:0/2292797903 learned_addr learned my addr 172.21.15.32:0/2292797903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.224+0000 7f319da83640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba90800 0x5639fba95af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319e284640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fbaff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 -- 172.21.15.32:0/2292797903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba90800 msgr2=0x5639fba95af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba90800 0x5639fba95af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 -- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 msgr2=0x5639fbaff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fbaff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 -- 172.21.15.32:0/2292797903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639fba92bf0 con 0x5639fba85620 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319e284640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fbaff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f319da83640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f318800a450 tx=0x7f318800a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e983093f0d9faadf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f3181ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188019070 con 0x5639fba85620 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f318800dac0 con 0x5639fba85620 2024-04-23T18:27:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.225+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318801a040 con 0x5639fba85620 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3188034040 con 0x5639fba85620 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 10 client.17013 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5639fbb06920 con 0x5639fba85620 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318801a040 con 0x5639fba85620 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f3188041020 con 0x5639fba85620 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 10 client.17013 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 20 client.17013 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbb06d90 0x5639fbb091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 4 client.17013 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639fb9f2200 con 0x5639fbb06d90 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.226+0000 7f319e284640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbb06d90 0x5639fbb091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.227+0000 7f3181ffb640 10 client.17013 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.227+0000 7f319e284640 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbb06d90 0x5639fbb091b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=187ec42df3089a13 server_cookie=a5f75bdd0d855c50 in_seq=0 out_seq=0 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.236+0000 7f3181ffb640 1 -- 172.21.15.32:0/2292797903 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639fb9f2200 con 0x5639fbb06d90 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.236+0000 7f3181ffb640 10 client.17013 handle_command_reply: tid=0 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.236+0000 7f31a3202740 1 client.17013 shutdown 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.236+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 msgr2=0x5639fbaff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.236+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f318800a450 tx=0x7f318800a380 comp rx=0 tx=0).stop 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 shutdown_connections 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5639fbb06d90 0x5639fbb091b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fba90800 0x5639fba95af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639fba85f40 0x5639fbaff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 --2- 172.21.15.32:0/2292797903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5639fba85620 0x5639fbaff330 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f3181ffb640 0 client.17013 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 >> 172.21.15.32:0/2292797903 conn(0x5639fba81b30 msgr2=0x5639fba80420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 shutdown_connections 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 1 -- 172.21.15.32:0/2292797903 wait complete. 2024-04-23T18:27:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.237+0000 7f31a3202740 20 client.17013 trim_cache size 0 max 16384 2024-04-23T18:27:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:27:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:27:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:27:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 msgr2=0x7f31840a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 0x7f31840a62d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31900cd1d0 tx=0x7f3190013a60 comp rx=0 tx=0).stop 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 msgr2=0x7f319819adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3190005ab0 tx=0x7f3190013660 comp rx=0 tx=0).stop 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 shutdown_connections 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f319810c590 0x7f319819adc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3198103ae0 0x7f3198196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f31840a3e10 0x7f31840a62d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.240+0000 7f31a04e8640 1 --2- 172.21.15.32:0/4266634993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31981031c0 0x7f3198196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.241+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 >> 172.21.15.32:0/4266634993 conn(0x7f3198077f70 msgr2=0x7f31980fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.241+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 shutdown_connections 2024-04-23T18:27:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:30.241+0000 7f31a04e8640 1 -- 172.21.15.32:0/4266634993 wait complete. 2024-04-23T18:27:30.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:29 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:30.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:29 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:30.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:29 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:27:31.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:30 smithi040 ceph-mon[32626]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 521 op/s 2024-04-23T18:27:31.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:30 smithi064 ceph-mon[33062]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 521 op/s 2024-04-23T18:27:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:30 smithi032 ceph-mon[29221]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 521 op/s 2024-04-23T18:27:33.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:32 smithi040 ceph-mon[32626]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 432 op/s 2024-04-23T18:27:33.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:32 smithi064 ceph-mon[33062]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 432 op/s 2024-04-23T18:27:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:32 smithi032 ceph-mon[29221]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 432 op/s 2024-04-23T18:27:33.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 725.0 s, 2772.4 tps, lat 36.022 ms stddev 41.989 2024-04-23T18:27:35.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:34 smithi040 ceph-mon[32626]: pgmap v389: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 438 op/s 2024-04-23T18:27:35.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:34 smithi064 ceph-mon[33062]: pgmap v389: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 438 op/s 2024-04-23T18:27:35.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:34 smithi032 ceph-mon[29221]: pgmap v389: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.6 MiB/s wr, 438 op/s 2024-04-23T18:27:37.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:36 smithi040 ceph-mon[32626]: pgmap v390: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.1 MiB/s wr, 749 op/s 2024-04-23T18:27:37.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:36 smithi064 ceph-mon[33062]: pgmap v390: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.1 MiB/s wr, 749 op/s 2024-04-23T18:27:37.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:36 smithi032 ceph-mon[29221]: pgmap v390: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.1 MiB/s wr, 749 op/s 2024-04-23T18:27:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 730.0 s, 2604.8 tps, lat 38.606 ms stddev 43.048 2024-04-23T18:27:39.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:38 smithi040 ceph-mon[32626]: pgmap v391: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-23T18:27:39.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:38 smithi064 ceph-mon[33062]: pgmap v391: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-23T18:27:39.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:38 smithi032 ceph-mon[29221]: pgmap v391: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-23T18:27:41.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:40 smithi040 ceph-mon[32626]: pgmap v392: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 938 op/s 2024-04-23T18:27:41.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:40 smithi064 ceph-mon[33062]: pgmap v392: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 938 op/s 2024-04-23T18:27:41.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:40 smithi032 ceph-mon[29221]: pgmap v392: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 938 op/s 2024-04-23T18:27:43.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:42 smithi040 ceph-mon[32626]: pgmap v393: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 641 op/s 2024-04-23T18:27:43.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:42 smithi064 ceph-mon[33062]: pgmap v393: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 641 op/s 2024-04-23T18:27:43.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:42 smithi032 ceph-mon[29221]: pgmap v393: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 641 op/s 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- 172.21.15.32:0/1557133694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 msgr2=0x7f8658115360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 --2- 172.21.15.32:0/1557133694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f8658115360 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ca0 tx=0x7f865002a950 comp rx=0 tx=0).stop 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- 172.21.15.32:0/1557133694 shutdown_connections 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 --2- 172.21.15.32:0/1557133694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f8658115360 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 --2- 172.21.15.32:0/1557133694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f8658112a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 --2- 172.21.15.32:0/1557133694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f86580746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- 172.21.15.32:0/1557133694 >> 172.21.15.32:0/1557133694 conn(0x7f86580fda30 msgr2=0x7f86580ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- 172.21.15.32:0/1557133694 shutdown_connections 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- 172.21.15.32:0/1557133694 wait complete. 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 Processor -- start 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.366+0000 7f865f4cf640 1 -- start start 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f865819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f865819f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86581187e0 con 0x7f8658112f70 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8658118660 con 0x7f8658074c10 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865f4cf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8658118960 con 0x7f86580742f0 2024-04-23T18:27:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865da6c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f865819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865da6c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54422/0 (socket says 172.21.15.32:54422) 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865da6c640 1 -- 172.21.15.32:0/620185357 learned_addr learned my addr 172.21.15.32:0/620185357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865ca6a640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f865819f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 -- 172.21.15.32:0/620185357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 msgr2=0x7f865819f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f865819f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 -- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 msgr2=0x7f86581a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 -- 172.21.15.32:0/620185357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86581a3d50 con 0x7f86580742f0 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865ca6a640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f865819f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865da6c640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.367+0000 7f865d26b640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f865819ed60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f864800e9f0 tx=0x7f864800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.368+0000 7f864e7fc640 1 -- 172.21.15.32:0/620185357 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800ccc0 con 0x7f86580742f0 2024-04-23T18:27:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.368+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86581a4040 con 0x7f86580742f0 2024-04-23T18:27:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.368+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86581ab8d0 con 0x7f86580742f0 2024-04-23T18:27:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.368+0000 7f864e7fc640 1 -- 172.21.15.32:0/620185357 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8648005020 con 0x7f86580742f0 2024-04-23T18:27:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.368+0000 7f864e7fc640 1 -- 172.21.15.32:0/620185357 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648004a00 con 0x7f86580742f0 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f864e7fc640 1 -- 172.21.15.32:0/620185357 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8648004ba0 con 0x7f86580742f0 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 Processor -- start 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 -- start start 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f864e7fc640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 0x7f86440a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 0x559f6f88e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f890bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f6f7e8970 con 0x559f6f883f40 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f6f7854a0 con 0x559f6f883620 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.371+0000 7f86621e9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559f6f7e4930 con 0x559f6f88e800 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865ca6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 0x559f6f88e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865da6c640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 0x7f86440a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:58324/0 (socket says 172.21.15.32:58324) 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865ca6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 0x559f6f88e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54430/0 (socket says 172.21.15.32:54430) 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 -- 172.21.15.32:0/1441795651 learned_addr learned my addr 172.21.15.32:0/1441795651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865da6c640 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f890bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f864e7fc640 1 -- 172.21.15.32:0/620185357 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f86480d0790 con 0x7f86580742f0 2024-04-23T18:27:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 -- 172.21.15.32:0/1441795651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 msgr2=0x559f6f890bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f890bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 -- 172.21.15.32:0/1441795651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 msgr2=0x559f6f88e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 0x559f6f88e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 -- 172.21.15.32:0/1441795651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f6f4fb210 con 0x559f6f883620 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865da6c640 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f890bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865d26b640 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f864800c6a0 tx=0x7f864809c050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d1539816493a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f86437fe640 1 -- 172.21.15.32:0/1441795651 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864801c070 con 0x559f6f883620 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.372+0000 7f865da6c640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 0x7f86440a6250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559f6f880f50 tx=0x7f86500093d0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.373+0000 7f86437fe640 1 -- 172.21.15.32:0/1441795651 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f864809cdf0 con 0x559f6f883620 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.373+0000 7f86437fe640 1 -- 172.21.15.32:0/1441795651 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480a5360 con 0x559f6f883620 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.373+0000 7f86621e9740 1 -- 172.21.15.32:0/1441795651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 msgr2=0x559f6f883a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.373+0000 7f86621e9740 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f864800c6a0 tx=0x7f864809c050 comp rx=0 tx=0).stop 2024-04-23T18:27:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 -- 172.21.15.32:0/1441795651 shutdown_connections 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f890bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883f40 0x559f6f88e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 --2- 172.21.15.32:0/1441795651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f883a00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 -- 172.21.15.32:0/1441795651 >> 172.21.15.32:0/1441795651 conn(0x559f6f87fb30 msgr2=0x559f6f87ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 -- 172.21.15.32:0/1441795651 shutdown_connections 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.374+0000 7f86621e9740 1 -- 172.21.15.32:0/1441795651 wait complete. 2024-04-23T18:27:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 -- start start 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 Processor -- start 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f88e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 0x559f6f88b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f88ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f6f8913d0 con 0x559f6f88e800 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f6f7830b0 con 0x559f6f883620 2024-04-23T18:27:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.375+0000 7f86621e9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559f6f891550 con 0x559f6f883f40 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f86621e9740 10 client.0 _ll_register_callbacks cb 0x559f6f770d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865ca6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 0x559f6f88b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f88ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865ca6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 0x559f6f88b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:57382/0 (socket says 172.21.15.32:57382) 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865ca6a640 1 -- 172.21.15.32:0/3511621255 learned_addr learned my addr 172.21.15.32:0/3511621255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865d26b640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f88e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 -- 172.21.15.32:0/3511621255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 msgr2=0x559f6f88b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 0x559f6f88b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 -- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 msgr2=0x559f6f88e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f88e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 -- 172.21.15.32:0/3511621255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f6f5c7620 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865d26b640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f88e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f865da6c640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f88ba00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x559f6f880cc0 tx=0x7f86500095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1378e99680b2bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f8640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.376+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650034040 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f8650004450 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f864809cdf0 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8650004650 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 10 client.17025 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559f6f89d290 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f864809cdf0 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f865004f070 con 0x559f6f88e800 2024-04-23T18:27:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 10 client.17025 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 20 client.17025 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f89e7c0 0x559f6f8a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 4 client.17025 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f6f7f6390 con 0x559f6f89e7c0 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.377+0000 7f865d26b640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f89e7c0 0x559f6f8a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.378+0000 7f865d26b640 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f89e7c0 0x559f6f8a0bb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9ca53debd6a273a server_cookie=4cc75eb137174784 in_seq=0 out_seq=0 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.378+0000 7f8640ff9640 10 client.17025 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.378+0000 7f8640ff9640 1 -- 172.21.15.32:0/3511621255 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f6f7f6390 con 0x559f6f89e7c0 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.378+0000 7f8640ff9640 10 client.17025 handle_command_reply: tid=0 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.378+0000 7f86621e9740 1 client.17025 shutdown 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 msgr2=0x559f6f88ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f88ba00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x559f6f880cc0 tx=0x7f86500095c0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 shutdown_connections 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f89e7c0 0x559f6f8a0bb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f88ba00 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883f40 0x559f6f88b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 --2- 172.21.15.32:0/3511621255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883620 0x559f6f88e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 >> 172.21.15.32:0/3511621255 conn(0x559f6f87fb30 msgr2=0x559f6f8858b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f8640ff9640 0 client.17025 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 shutdown_connections 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 1 -- 172.21.15.32:0/3511621255 wait complete. 2024-04-23T18:27:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.379+0000 7f86621e9740 20 client.17025 trim_cache size 0 max 16384 2024-04-23T18:27:43.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 Processor -- start 2024-04-23T18:27:43.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 -- start start 2024-04-23T18:27:43.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f87eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f8a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86581ae370 con 0x559f6f883620 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f86581abbb0 con 0x559f6f883f40 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f86621e9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f865810c8c0 con 0x559f6f88e800 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f865d26b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f865ca6a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f87eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f865d26b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54434/0 (socket says 172.21.15.32:54434) 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.401+0000 7f865d26b640 1 -- 172.21.15.32:0/2155568975 learned_addr learned my addr 172.21.15.32:0/2155568975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865da6c640 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f8a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 -- 172.21.15.32:0/2155568975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 msgr2=0x559f6f8a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f8a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 -- 172.21.15.32:0/2155568975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 msgr2=0x559f6f87eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f87eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 -- 172.21.15.32:0/2155568975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f6f87b2e0 con 0x559f6f883620 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865ca6a640 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f87eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f865d26b640 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f86480a4e30 tx=0x7f8648087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3ef2b95669e361 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f8642ffd640 1 -- 172.21.15.32:0/2155568975 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864801c040 con 0x559f6f883620 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f8642ffd640 1 -- 172.21.15.32:0/2155568975 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86480a5360 con 0x559f6f883620 2024-04-23T18:27:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.402+0000 7f8642ffd640 1 -- 172.21.15.32:0/2155568975 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480ce120 con 0x559f6f883620 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- 172.21.15.32:0/2155568975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 msgr2=0x559f6f89a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f86480a4e30 tx=0x7f8648087040 comp rx=0 tx=0).stop 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- 172.21.15.32:0/2155568975 shutdown_connections 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f88e800 0x559f6f8a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f87eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 --2- 172.21.15.32:0/2155568975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f883620 0x559f6f89a290 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- 172.21.15.32:0/2155568975 >> 172.21.15.32:0/2155568975 conn(0x559f6f87fb30 msgr2=0x559f6f87e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.401 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- 172.21.15.32:0/2155568975 shutdown_connections 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- 172.21.15.32:0/2155568975 wait complete. 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.403+0000 7f86621e9740 1 -- start start 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 Processor -- start 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f8fd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f893b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f865810cf10 con 0x559f6f88e800 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f865810e380 con 0x559f6f883f40 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.404+0000 7f86621e9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f865810d5e0 con 0x559f6f883620 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:57404/0 (socket says 172.21.15.32:57404) 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 -- 172.21.15.32:0/1195791871 learned_addr learned my addr 172.21.15.32:0/1195791871 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:43.402 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f86621e9740 10 client.0 _ll_register_callbacks cb 0x7f865002f760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865ca6a640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f8fd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865da6c640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f893b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 -- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 msgr2=0x559f6f8fd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f8fd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 -- 172.21.15.32:0/1195791871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 msgr2=0x559f6f893b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f893b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 -- 172.21.15.32:0/1195791871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f6f894270 con 0x559f6f883620 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865da6c640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f893b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865ca6a640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f8fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f865d26b640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f86480ae720 tx=0x7f8648005460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3753cf236139c6d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.405+0000 7f8640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480ba060 con 0x559f6f883620 2024-04-23T18:27:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f86480ce120 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86480a5360 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f864801c040 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 10 client.26545 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559f6f9048f0 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86480a5360 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f86480a4ed0 con 0x559f6f883620 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 10 client.26545 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 10 client.26545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 20 client.26545 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f904b70 0x559f6f906f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 4 client.26545 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f6f7f0200 con 0x559f6f904b70 2024-04-23T18:27:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.406+0000 7f865ca6a640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f904b70 0x559f6f906f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.407+0000 7f8640ff9640 10 client.26545 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.407+0000 7f865ca6a640 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f904b70 0x559f6f906f60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8038275dcc88ffe server_cookie=54983d765b76c520 in_seq=0 out_seq=0 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.415+0000 7f8640ff9640 1 -- 172.21.15.32:0/1195791871 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f6f7f0200 con 0x559f6f904b70 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.415+0000 7f8640ff9640 10 client.26545 handle_command_reply: tid=0 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.415+0000 7f86621e9740 1 client.26545 shutdown 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 msgr2=0x559f6f8fd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f86480ae720 tx=0x7f8648005460 comp rx=0 tx=0).stop 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 shutdown_connections 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x559f6f904b70 0x559f6f906f60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f6f88e800 0x559f6f893b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f6f883f40 0x559f6f8fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f8640ff9640 0 client.26545 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 --2- 172.21.15.32:0/1195791871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559f6f883620 0x559f6f8fd100 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 >> 172.21.15.32:0/1195791871 conn(0x559f6f87fb30 msgr2=0x559f6f87ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 shutdown_connections 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 1 -- 172.21.15.32:0/1195791871 wait complete. 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f86621e9740 20 client.26545 trim_cache size 0 max 16384 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:27:43.410 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:27:43.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 msgr2=0x7f86440a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 0x7f86440a6250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559f6f880f50 tx=0x7f86500093d0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.411 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 msgr2=0x7f865819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.416+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f865819ed60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f864800e9f0 tx=0x7f864800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 shutdown_connections 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f86440a3d90 0x7f86440a6250 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8658112f70 0x7f86581a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658074c10 0x7f865819f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 --2- 172.21.15.32:0/620185357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580742f0 0x7f865819ed60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 >> 172.21.15.32:0/620185357 conn(0x7f86580fda30 msgr2=0x7f86580ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 shutdown_connections 2024-04-23T18:27:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:43.417+0000 7f865f4cf640 1 -- 172.21.15.32:0/620185357 wait complete. 2024-04-23T18:27:43.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 735.0 s, 1793.6 tps, lat 55.817 ms stddev 69.070 2024-04-23T18:27:45.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:44 smithi040 ceph-mon[32626]: pgmap v394: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-04-23T18:27:45.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:44 smithi064 ceph-mon[33062]: pgmap v394: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-04-23T18:27:45.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:44 smithi032 ceph-mon[29221]: pgmap v394: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-04-23T18:27:47.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:46 smithi040 ceph-mon[32626]: pgmap v395: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-23T18:27:47.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:46 smithi064 ceph-mon[33062]: pgmap v395: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-23T18:27:47.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:46 smithi032 ceph-mon[29221]: pgmap v395: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-23T18:27:48.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 740.0 s, 1869.0 tps, lat 53.638 ms stddev 101.418 2024-04-23T18:27:49.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:48 smithi040 ceph-mon[32626]: pgmap v396: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.6 MiB/s wr, 566 op/s 2024-04-23T18:27:49.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:48 smithi064 ceph-mon[33062]: pgmap v396: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.6 MiB/s wr, 566 op/s 2024-04-23T18:27:49.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:48 smithi032 ceph-mon[29221]: pgmap v396: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.6 MiB/s wr, 566 op/s 2024-04-23T18:27:51.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:50 smithi040 ceph-mon[32626]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:27:51.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:50 smithi064 ceph-mon[33062]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:27:51.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:50 smithi032 ceph-mon[29221]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:27:53.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:52 smithi040 ceph-mon[32626]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.65k op/s 2024-04-23T18:27:53.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:52 smithi064 ceph-mon[33062]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.65k op/s 2024-04-23T18:27:53.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:52 smithi032 ceph-mon[29221]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.65k op/s 2024-04-23T18:27:53.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 745.0 s, 2517.8 tps, lat 39.674 ms stddev 44.780 2024-04-23T18:27:55.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:54 smithi040 ceph-mon[32626]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.66k op/s 2024-04-23T18:27:55.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:54 smithi064 ceph-mon[33062]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.66k op/s 2024-04-23T18:27:55.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:54 smithi032 ceph-mon[29221]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 MiB/s wr, 1.66k op/s 2024-04-23T18:27:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.538+0000 7f5552ffd640 1 -- 172.21.15.32:0/3474618332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c02fd10 con 0x7f55541063b0 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 -- 172.21.15.32:0/3474618332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55541063b0 msgr2=0x7f5554106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 --2- 172.21.15.32:0/3474618332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55541063b0 0x7f5554106860 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f554c009d00 tx=0x7f554c02a9a0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 -- 172.21.15.32:0/3474618332 shutdown_connections 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 --2- 172.21.15.32:0/3474618332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55541063b0 0x7f5554106860 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 --2- 172.21.15.32:0/3474618332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541018d0 0x7f5554105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 --2- 172.21.15.32:0/3474618332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5554100fb0 0x7f5554101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 -- 172.21.15.32:0/3474618332 >> 172.21.15.32:0/3474618332 conn(0x7f5554077fb0 msgr2=0x7f55540783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 -- 172.21.15.32:0/3474618332 shutdown_connections 2024-04-23T18:27:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 -- 172.21.15.32:0/3474618332 wait complete. 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.539+0000 7f555a744640 1 Processor -- start 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 -- start start 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 0x7f55541a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55541018d0 0x7f55541a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 0x7f555419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5554118890 con 0x7f5554100fb0 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5554118710 con 0x7f55541063b0 2024-04-23T18:27:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f555a744640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5554118a10 con 0x7f55541018d0 2024-04-23T18:27:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f5553fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 0x7f55541a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f5553fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 0x7f55541a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54184/0 (socket says 172.21.15.32:54184) 2024-04-23T18:27:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f5553fff640 1 -- 172.21.15.32:0/356361554 learned_addr learned my addr 172.21.15.32:0/356361554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f5558ce1640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 0x7f555419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.540+0000 7f55537fe640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55541018d0 0x7f55541a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 -- 172.21.15.32:0/356361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55541018d0 msgr2=0x7f55541a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55541018d0 0x7f55541a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 -- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 msgr2=0x7f55541a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 0x7f55541a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 -- 172.21.15.32:0/356361554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555419f420 con 0x7f55541063b0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f5558ce1640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 0x7f555419eb60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f554c02aec0 tx=0x7f554c02feb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f55517fa640 1 -- 172.21.15.32:0/356361554 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c034030 con 0x7f55541063b0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555419f6b0 con 0x7f55541063b0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f55517fa640 1 -- 172.21.15.32:0/356361554 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f554c030b10 con 0x7f55541063b0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f55517fa640 1 -- 172.21.15.32:0/356361554 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c004760 con 0x7f55541063b0 2024-04-23T18:27:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.541+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55541ab7f0 con 0x7f55541063b0 2024-04-23T18:27:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.545+0000 7f55517fa640 1 -- 172.21.15.32:0/356361554 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f554c037020 con 0x7f55541063b0 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.545+0000 7f555d465740 1 Processor -- start 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 -- start start 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f55517fa640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 0x7f553c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fd82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f55517fa640 1 -- 172.21.15.32:0/356361554 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f554c0f13a0 con 0x7f55541063b0 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd8810 0x562de3fdac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562de3f21970 con 0x562de3fbce00 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562de3ebe4a0 con 0x562de3fd8810 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f555d465740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562de3f1d930 con 0x562de3fbc4e0 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f5553fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f55537fe640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 0x7f553c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f5553fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:59788/0 (socket says 172.21.15.32:59788) 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f5553fff640 1 -- 172.21.15.32:0/300033472 learned_addr learned my addr 172.21.15.32:0/300033472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.546+0000 7f5558ce1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd8810 0x562de3fdac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 -- 172.21.15.32:0/300033472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd8810 msgr2=0x562de3fdac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd8810 0x562de3fdac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 -- 172.21.15.32:0/300033472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 msgr2=0x562de3fd82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fd82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 -- 172.21.15.32:0/300033472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562de3c34210 con 0x562de3fbc4e0 2024-04-23T18:27:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f5553fff640 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f554800bdd0 tx=0x7f55480107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53505d84b5177381 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f553a7fc640 1 -- 172.21.15.32:0/300033472 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548005020 con 0x562de3fbc4e0 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f55537fe640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 0x7f553c0a61d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x562de3fbdba0 tx=0x7f55440094e0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.547+0000 7f553a7fc640 1 -- 172.21.15.32:0/300033472 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55480047a0 con 0x562de3fbc4e0 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f553a7fc640 1 -- 172.21.15.32:0/300033472 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55480116e0 con 0x562de3fbc4e0 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 -- 172.21.15.32:0/300033472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 msgr2=0x562de3fbc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f554800bdd0 tx=0x7f55480107a0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 -- 172.21.15.32:0/300033472 shutdown_connections 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd8810 0x562de3fdac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fd82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 --2- 172.21.15.32:0/300033472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fbc8c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 -- 172.21.15.32:0/300033472 >> 172.21.15.32:0/300033472 conn(0x562de3fb8b30 msgr2=0x562de3fb8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 -- 172.21.15.32:0/300033472 shutdown_connections 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.548+0000 7f555d465740 1 -- 172.21.15.32:0/300033472 wait complete. 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 -- start start 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 Processor -- start 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fd8810 0x562de3fc4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562de3fbf4f0 con 0x562de3fd8810 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562de3ebc0b0 con 0x562de3fbc4e0 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562de3fbf670 con 0x562de3fbce00 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f55537fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f5553fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f5553fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:50234/0 (socket says 172.21.15.32:50234) 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f55537fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:59802/0 (socket says 172.21.15.32:59802) 2024-04-23T18:27:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f5553fff640 1 -- 172.21.15.32:0/1462326961 learned_addr learned my addr 172.21.15.32:0/1462326961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.549+0000 7f555d465740 10 client.0 _ll_register_callbacks cb 0x562de3ea9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5558ce1640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fd8810 0x562de3fc4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 -- 172.21.15.32:0/1462326961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 msgr2=0x562de3fc9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 -- 172.21.15.32:0/1462326961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fd8810 msgr2=0x562de3fc4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fd8810 0x562de3fc4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 -- 172.21.15.32:0/1462326961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562de3d00620 con 0x562de3fbc4e0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f55537fe640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f5553fff640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5548011010 tx=0x7f5548004b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd612128b241479 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f55337fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548029060 con 0x562de3fbc4e0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f55480051f0 con 0x562de3fbc4e0 2024-04-23T18:27:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55480047a0 con 0x562de3fbc4e0 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5548004ca0 con 0x562de3fbc4e0 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f555d465740 10 client.26540 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.550+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562de3fd39b0 con 0x562de3fbc4e0 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.551+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55480047a0 con 0x562de3fbc4e0 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f554801b430 con 0x562de3fbc4e0 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 10 client.26540 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 10 client.26540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 20 client.26540 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd4e10 0x562de3fd72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 4 client.26540 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562de3f2f390 con 0x562de3fd4e10 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.552+0000 7f55537fe640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd4e10 0x562de3fd72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.553+0000 7f55537fe640 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd4e10 0x562de3fd72d0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559152e1996cb80d server_cookie=5f6823205e15315b in_seq=0 out_seq=0 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.553+0000 7f55337fe640 10 client.26540 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f55337fe640 1 -- 172.21.15.32:0/1462326961 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562de3f2f390 con 0x562de3fd4e10 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f55337fe640 10 client.26540 handle_command_reply: tid=0 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 client.26540 shutdown 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 msgr2=0x562de3fc9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5548011010 tx=0x7f5548004b80 comp rx=0 tx=0).stop 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 shutdown_connections 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd4e10 0x562de3fd72d0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f55337fe640 0 client.26540 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fd8810 0x562de3fc4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de3fc9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 --2- 172.21.15.32:0/1462326961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fbc4e0 0x562de3fc9780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.554+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 >> 172.21.15.32:0/1462326961 conn(0x562de3fb8b30 msgr2=0x562de3fbe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.555+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 shutdown_connections 2024-04-23T18:27:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.555+0000 7f555d465740 1 -- 172.21.15.32:0/1462326961 wait complete. 2024-04-23T18:27:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.555+0000 7f555d465740 20 client.26540 trim_cache size 0 max 16384 2024-04-23T18:27:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 Processor -- start 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 -- start start 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbc4e0 0x562de3fd8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de4002720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f55541ae240 con 0x562de3fbc4e0 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55541abab0 con 0x562de3fd4e10 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f555d465740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f555419ffd0 con 0x562de3fbce00 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:59816/0 (socket says 172.21.15.32:59816) 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 -- 172.21.15.32:0/3169540401 learned_addr learned my addr 172.21.15.32:0/3169540401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f5553fff640 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbc4e0 0x562de3fd8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f5558ce1640 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de4002720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 -- 172.21.15.32:0/3169540401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 msgr2=0x562de4002720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de4002720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 -- 172.21.15.32:0/3169540401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbc4e0 msgr2=0x562de3fd8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbc4e0 0x562de3fd8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.579+0000 7f55537fe640 1 -- 172.21.15.32:0/3169540401 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562de3fb8960 con 0x562de3fbce00 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.580+0000 7f55537fe640 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f55541a02f0 tx=0x7f5544012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402fdbec8b60e04d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.580+0000 7f5539ffb640 1 -- 172.21.15.32:0/3169540401 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554402f070 con 0x562de3fbce00 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.580+0000 7f5539ffb640 1 -- 172.21.15.32:0/3169540401 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5544003770 con 0x562de3fbce00 2024-04-23T18:27:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.580+0000 7f5539ffb640 1 -- 172.21.15.32:0/3169540401 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544010690 con 0x562de3fbce00 2024-04-23T18:27:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.580+0000 7f555d465740 1 -- 172.21.15.32:0/3169540401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 msgr2=0x562de40021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f55541a02f0 tx=0x7f5544012000 comp rx=0 tx=0).stop 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 -- 172.21.15.32:0/3169540401 shutdown_connections 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de4002720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbce00 0x562de40021e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 --2- 172.21.15.32:0/3169540401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbc4e0 0x562de3fd8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 -- 172.21.15.32:0/3169540401 >> 172.21.15.32:0/3169540401 conn(0x562de3fb8b30 msgr2=0x562de3fc0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 -- 172.21.15.32:0/3169540401 shutdown_connections 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 -- 172.21.15.32:0/3169540401 wait complete. 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 -- start start 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.581+0000 7f555d465740 1 Processor -- start 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fc1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de3fcf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f555410f300 con 0x562de3fbce00 2024-04-23T18:27:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5554110730 con 0x562de3fd4e10 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f555410f990 con 0x562de3fbc4e0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f555d465740 10 client.0 _ll_register_callbacks cb 0x7f554800be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f5553fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f5553fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:59818/0 (socket says 172.21.15.32:59818) 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f5553fff640 1 -- 172.21.15.32:0/3427644336 learned_addr learned my addr 172.21.15.32:0/3427644336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f55537fe640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fc1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.582+0000 7f5558ce1640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de3fcf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 -- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 msgr2=0x562de3fcf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de3fcf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 -- 172.21.15.32:0/3427644336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 msgr2=0x562de3fc1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fc1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 -- 172.21.15.32:0/3427644336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562de3fcfa50 con 0x562de3fbc4e0 2024-04-23T18:27:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f5553fff640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5548000c00 tx=0x7f554801b640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4dfb0dca036ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f55337fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548029070 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8484+0+0 (secure 0 0 0) 0x7f5548005400 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5544003770 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5548004e40 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f555d465740 10 client.26563 fetch_fsmap learned FSMap version 44 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562de405f2b0 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.583+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5544003770 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f5548035070 con 0x562de3fbc4e0 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 10 client.26563 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 10 client.26563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 20 client.26563 populate_metadata read hostname 'smithi032' 2024-04-23T18:27:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd8810 0x562de4061a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:56.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 4 client.26563 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:56.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562de3f29200 con 0x562de3fd8810 2024-04-23T18:27:56.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f55537fe640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd8810 0x562de4061a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:56.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f55537fe640 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd8810 0x562de4061a70 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b251d35eead37a90 server_cookie=ba6e624f6039e5e7 in_seq=0 out_seq=0 2024-04-23T18:27:56.582 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.584+0000 7f55337fe640 10 client.26563 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.592+0000 7f55337fe640 1 -- 172.21.15.32:0/3427644336 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562de3f29200 con 0x562de3fd8810 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.592+0000 7f55337fe640 10 client.26563 handle_command_reply: tid=0 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.592+0000 7f555d465740 1 client.26563 shutdown 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 msgr2=0x562de3fc17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5548000c00 tx=0x7f554801b640 comp rx=0 tx=0).stop 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 shutdown_connections 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x562de3fd8810 0x562de4061a70 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562de3fd4e10 0x562de3fcf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f55337fe640 0 client.26563 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562de3fbce00 0x562de3fc1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 --2- 172.21.15.32:0/3427644336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562de3fbc4e0 0x562de3fc17c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 >> 172.21.15.32:0/3427644336 conn(0x562de3fb8b30 msgr2=0x562de3fc0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 shutdown_connections 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 1 -- 172.21.15.32:0/3427644336 wait complete. 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555d465740 20 client.26563 trim_cache size 0 max 16384 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:27:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.593+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 msgr2=0x7f553c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 0x7f553c0a61d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x562de3fbdba0 tx=0x7f55440094e0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 msgr2=0x7f555419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 0x7f555419eb60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f554c02aec0 tx=0x7f554c02feb0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 shutdown_connections 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55541063b0 0x7f555419eb60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55541018d0 0x7f55541a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f553c0a3d10 0x7f553c0a61d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.590 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 --2- 172.21.15.32:0/356361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554100fb0 0x7f55541a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:56.590 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 >> 172.21.15.32:0/356361554 conn(0x7f5554077fb0 msgr2=0x7f5554103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:56.590 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 shutdown_connections 2024-04-23T18:27:56.590 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:27:56.594+0000 7f555a744640 1 -- 172.21.15.32:0/356361554 wait complete. 2024-04-23T18:27:57.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:56 smithi040 ceph-mon[32626]: pgmap v400: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 1.92k op/s 2024-04-23T18:27:57.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:56 smithi064 ceph-mon[33062]: pgmap v400: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 1.92k op/s 2024-04-23T18:27:57.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:56 smithi032 ceph-mon[29221]: pgmap v400: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 1.92k op/s 2024-04-23T18:27:58.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 750.0 s, 2637.0 tps, lat 37.542 ms stddev 42.660 2024-04-23T18:27:59.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:27:58 smithi040 ceph-mon[32626]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.67k op/s 2024-04-23T18:27:59.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:27:58 smithi064 ceph-mon[33062]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.67k op/s 2024-04-23T18:27:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:27:58 smithi032 ceph-mon[29221]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.67k op/s 2024-04-23T18:28:01.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:00 smithi040 ceph-mon[32626]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:28:01.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:00 smithi064 ceph-mon[33062]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:28:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:00 smithi032 ceph-mon[29221]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 MiB/s wr, 1.96k op/s 2024-04-23T18:28:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:02 smithi064 ceph-mon[33062]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 564 op/s 2024-04-23T18:28:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:02 smithi032 ceph-mon[29221]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 564 op/s 2024-04-23T18:28:03.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:02 smithi040 ceph-mon[32626]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 564 op/s 2024-04-23T18:28:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 755.0 s, 2124.6 tps, lat 47.265 ms stddev 106.491 2024-04-23T18:28:05.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:04 smithi064 ceph-mon[33062]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-04-23T18:28:05.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:04 smithi032 ceph-mon[29221]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-04-23T18:28:05.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:04 smithi040 ceph-mon[32626]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-04-23T18:28:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:06 smithi064 ceph-mon[33062]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.92k op/s 2024-04-23T18:28:07.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:06 smithi064 ceph-mon[33062]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:28:07.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:06 smithi032 ceph-mon[29221]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.92k op/s 2024-04-23T18:28:07.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:06 smithi032 ceph-mon[29221]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:28:07.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:06 smithi040 ceph-mon[32626]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.92k op/s 2024-04-23T18:28:07.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:06 smithi040 ceph-mon[32626]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:28:08.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 760.0 s, 2180.4 tps, lat 46.054 ms stddev 61.459 2024-04-23T18:28:09.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:08 smithi064 ceph-mon[33062]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.98k op/s 2024-04-23T18:28:09.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:08 smithi032 ceph-mon[29221]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.98k op/s 2024-04-23T18:28:09.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:08 smithi040 ceph-mon[32626]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.98k op/s 2024-04-23T18:28:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 -- 172.21.15.32:0/547934189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0102b60 msgr2=0x7f1fc0106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 --2- 172.21.15.32:0/547934189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0102b60 0x7f1fc0106ba0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009900 tx=0x7f1fb402a3f0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 -- 172.21.15.32:0/547934189 shutdown_connections 2024-04-23T18:28:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 --2- 172.21.15.32:0/547934189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0107280 0x7f1fc010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 --2- 172.21.15.32:0/547934189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0102b60 0x7f1fc0106ba0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 --2- 172.21.15.32:0/547934189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01021b0 0x7f1fc0102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 -- 172.21.15.32:0/547934189 >> 172.21.15.32:0/547934189 conn(0x7f1fc00fda30 msgr2=0x7f1fc00ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.742+0000 7f1fc5672640 1 -- 172.21.15.32:0/547934189 shutdown_connections 2024-04-23T18:28:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.743+0000 7f1fc5672640 1 -- 172.21.15.32:0/547934189 wait complete. 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.743+0000 7f1fc5672640 1 Processor -- start 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 -- start start 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0107280 0x7f1fc010e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01a36b0 0x7f1fc01a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc01121c0 con 0x7f1fc01a36b0 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc0112040 con 0x7f1fc01021b0 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fc5672640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc0112340 con 0x7f1fc0107280 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fbe7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0107280 0x7f1fc010e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45966/0 (socket says 172.21.15.32:45966) 2024-04-23T18:28:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.744+0000 7f1fbeffd640 1 -- 172.21.15.32:0/1905649717 learned_addr learned my addr 172.21.15.32:0/1905649717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01a36b0 0x7f1fc01a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 -- 172.21.15.32:0/1905649717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0107280 msgr2=0x7f1fc010e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0107280 0x7f1fc010e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 -- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01a36b0 msgr2=0x7f1fc01a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01a36b0 0x7f1fc01a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 -- 172.21.15.32:0/1905649717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc01a41a0 con 0x7f1fc01021b0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9f0 tx=0x7f1fb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fa7fff640 1 -- 172.21.15.32:0/1905649717 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000cd80 con 0x7f1fc01021b0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc01a4490 con 0x7f1fc01021b0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc01abca0 con 0x7f1fc01021b0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fa7fff640 1 -- 172.21.15.32:0/1905649717 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb00108e0 con 0x7f1fc01021b0 2024-04-23T18:28:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.745+0000 7f1fa7fff640 1 -- 172.21.15.32:0/1905649717 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0018980 con 0x7f1fc01021b0 2024-04-23T18:28:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.748+0000 7f1fa7fff640 1 -- 172.21.15.32:0/1905649717 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1fb0018b20 con 0x7f1fc01021b0 2024-04-23T18:28:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.749+0000 7f1fc8387740 1 Processor -- start 2024-04-23T18:28:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.749+0000 7f1fc8387740 1 -- start start 2024-04-23T18:28:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.749+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 0x5604c8751a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.749+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c875c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fc8387740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5604c86b6970 con 0x5604c875c800 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fc8387740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604c86534a0 con 0x5604c8751f40 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fc8387740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5604c86b2930 con 0x5604c8751620 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fbe7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fbe7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44336/0 (socket says 172.21.15.32:44336) 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/3232189944 learned_addr learned my addr 172.21.15.32:0/3232189944 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.750+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 0x5604c8751a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c875c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbf7fe640 1 -- 172.21.15.32:0/3232189944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 msgr2=0x5604c875c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c875c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fa7fff640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 0x7f1fa80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbf7fe640 1 -- 172.21.15.32:0/3232189944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 msgr2=0x5604c875ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbf7fe640 1 -- 172.21.15.32:0/3232189944 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c83c9210 con 0x5604c8751620 2024-04-23T18:28:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c875c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.751+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.752+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 0x5604c8751a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009e50 tx=0x7f1fb80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cbfad3fa510961d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.752+0000 7f1fa7fff640 1 -- 172.21.15.32:0/1905649717 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8528+0+0 (secure 0 0 0) 0x7f1fb00d0a70 con 0x7f1fc01021b0 2024-04-23T18:28:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.753+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 0x7f1fa80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.753+0000 7f1fa57fa640 1 -- 172.21.15.32:0/3232189944 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003cd0 con 0x5604c8751620 2024-04-23T18:28:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.753+0000 7f1fa57fa640 1 -- 172.21.15.32:0/3232189944 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb8015070 con 0x5604c8751620 2024-04-23T18:28:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.753+0000 7f1fa57fa640 1 -- 172.21.15.32:0/3232189944 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb801fb40 con 0x5604c8751620 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 0x7f1fa80a6250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fb801ffd0 tx=0x7f1fb801fdc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 -- 172.21.15.32:0/3232189944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 msgr2=0x5604c8751a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 0x5604c8751a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009e50 tx=0x7f1fb80107a0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 -- 172.21.15.32:0/3232189944 shutdown_connections 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c875c800 0x5604c875ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c875c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 --2- 172.21.15.32:0/3232189944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c8751620 0x5604c8751a00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 -- 172.21.15.32:0/3232189944 >> 172.21.15.32:0/3232189944 conn(0x5604c874db30 msgr2=0x5604c874df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 -- 172.21.15.32:0/3232189944 shutdown_connections 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.754+0000 7f1fc8387740 1 -- 172.21.15.32:0/3232189944 wait complete. 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 -- start start 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 Processor -- start 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751620 0x5604c875c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 0x5604c8759930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5604c8770440 con 0x5604c8751620 2024-04-23T18:28:09.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604c86510b0 con 0x5604c8751f40 2024-04-23T18:28:09.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5604c87705c0 con 0x5604c875c800 2024-04-23T18:28:09.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.755+0000 7f1fc8387740 10 client.0 _ll_register_callbacks cb 0x5604c863ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:45974/0 (socket says 172.21.15.32:45974) 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/4170372976 learned_addr learned my addr 172.21.15.32:0/4170372976 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 0x5604c8759930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751620 0x5604c875c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/4170372976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 msgr2=0x5604c8759930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 0x5604c8759930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/4170372976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751620 msgr2=0x5604c875c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751620 0x5604c875c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/4170372976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c8495620 con 0x5604c8751f40 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 0x5604c8759930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.756+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006010 tx=0x7f1fb4032e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a631e14b90b7bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fa2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4035040 con 0x5604c8751f40 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8528+0+0 (secure 0 0 0) 0x7f1fb4031760 con 0x5604c8751f40 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb8015070 con 0x5604c8751f40 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1fb4004040 con 0x5604c8751f40 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fc8387740 10 client.26555 fetch_fsmap learned FSMap version 44 2024-04-23T18:28:09.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.757+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5604c877c350 con 0x5604c8751f40 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb8015070 con 0x5604c8751f40 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f1fb4004860 con 0x5604c8751f40 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 10 client.26555 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 10 client.26555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 20 client.26555 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c877d830 0x5604c877fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 4 client.26555 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.758+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604c86c4390 con 0x5604c877d830 2024-04-23T18:28:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.759+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c877d830 0x5604c877fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.759+0000 7f1fa2ff5640 10 client.26555 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.759+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c877d830 0x5604c877fca0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c80cf5b5f57e864d server_cookie=183042172fc57bc2 in_seq=0 out_seq=0 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/4170372976 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604c86c4390 con 0x5604c877d830 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fa2ff5640 10 client.26555 handle_command_reply: tid=0 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 client.26555 shutdown 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 msgr2=0x5604c87593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006010 tx=0x7f1fb4032e90 comp rx=0 tx=0).stop 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 shutdown_connections 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c877d830 0x5604c877fca0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c875c800 0x5604c8759930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751f40 0x5604c87593f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 --2- 172.21.15.32:0/4170372976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751620 0x5604c875c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 >> 172.21.15.32:0/4170372976 conn(0x5604c874db30 msgr2=0x5604c8753860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fa2ff5640 0 client.26555 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 shutdown_connections 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 1 -- 172.21.15.32:0/4170372976 wait complete. 2024-04-23T18:28:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.760+0000 7f1fc8387740 20 client.26555 trim_cache size 0 max 16384 2024-04-23T18:28:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.786+0000 7f1fc8387740 1 Processor -- start 2024-04-23T18:28:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 -- start start 2024-04-23T18:28:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c875bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8785600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac0110d0 con 0x5604c8751f40 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac00e910 con 0x5604c8751620 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fc8387740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac00c760 con 0x5604c874db30 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8785600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:50020/0 (socket says 172.21.15.32:50020) 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbeffd640 1 -- 172.21.15.32:0/722310895 learned_addr learned my addr 172.21.15.32:0/722310895 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c875bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/722310895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 msgr2=0x5604c87536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/722310895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 msgr2=0x5604c8785600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8785600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbe7fc640 1 -- 172.21.15.32:0/722310895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c8759f30 con 0x5604c8751620 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbf7fe640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8785600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.787+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.788+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c875bc30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009900 tx=0x7f1fb402a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483cc42cf9f50cb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.788+0000 7f1fa4ff9640 1 -- 172.21.15.32:0/722310895 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4035030 con 0x5604c8751620 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.788+0000 7f1fa4ff9640 1 -- 172.21.15.32:0/722310895 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb4031760 con 0x5604c8751620 2024-04-23T18:28:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.788+0000 7f1fa4ff9640 1 -- 172.21.15.32:0/722310895 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4003a60 con 0x5604c8751620 2024-04-23T18:28:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 -- 172.21.15.32:0/722310895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 msgr2=0x5604c875bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c875bc30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009900 tx=0x7f1fb402a3f0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 -- 172.21.15.32:0/722310895 shutdown_connections 2024-04-23T18:28:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8785600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c875bc30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 --2- 172.21.15.32:0/722310895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c87536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.789+0000 7f1fc8387740 1 -- 172.21.15.32:0/722310895 >> 172.21.15.32:0/722310895 conn(0x5604c8759a90 msgr2=0x5604c8752fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.790+0000 7f1fc8387740 1 -- 172.21.15.32:0/722310895 shutdown_connections 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.790+0000 7f1fc8387740 1 -- 172.21.15.32:0/722310895 wait complete. 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 -- start start 2024-04-23T18:28:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 Processor -- start 2024-04-23T18:28:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c8771070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8771680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac00cdd0 con 0x5604c8751f40 2024-04-23T18:28:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac00e240 con 0x5604c8751620 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac00d4a0 con 0x5604c874db30 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fc8387740 10 client.0 _ll_register_callbacks cb 0x7f1fb4032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fbf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8771680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fbeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:50022/0 (socket says 172.21.15.32:50022) 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fbeffd640 1 -- 172.21.15.32:0/493490182 learned_addr learned my addr 172.21.15.32:0/493490182 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.791+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c8771070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 -- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 msgr2=0x5604c8771070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c8771070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 -- 172.21.15.32:0/493490182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 msgr2=0x5604c8771680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8771680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 -- 172.21.15.32:0/493490182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c8771e80 con 0x5604c874db30 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fbeffd640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c6a0 tx=0x7f1fb00d2f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a988ba25f8c9fc13 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fa2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:28:09.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb009e150 con 0x5604c874db30 2024-04-23T18:28:09.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.792+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8528+0+0 (secure 0 0 0) 0x7f1fb00ce440 con 0x5604c874db30 2024-04-23T18:28:09.787 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.793+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb4031760 con 0x5604c874db30 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.793+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1fb00ce640 con 0x5604c874db30 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.793+0000 7f1fc8387740 10 client.26581 fetch_fsmap learned FSMap version 44 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.793+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5604c87e3f50 con 0x5604c874db30 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.794+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb4031760 con 0x5604c874db30 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.794+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7f1fb00a51f0 con 0x5604c874db30 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 10 client.26581 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 10 client.26581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 20 client.26581 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c875c800 0x5604c87e6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 4 client.26581 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.795+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604c86be200 con 0x5604c875c800 2024-04-23T18:28:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.796+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c875c800 0x5604c87e6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.796+0000 7f1fbe7fc640 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c875c800 0x5604c87e6650 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=884563c7e6208723 server_cookie=db3bc06cf2c0d7e0 in_seq=0 out_seq=0 2024-04-23T18:28:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.796+0000 7f1fa2ff5640 10 client.26581 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:09.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.808+0000 7f1fa2ff5640 1 -- 172.21.15.32:0/493490182 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604c86be200 con 0x5604c875c800 2024-04-23T18:28:09.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.808+0000 7f1fa2ff5640 10 client.26581 handle_command_reply: tid=0 2024-04-23T18:28:09.801 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.808+0000 7f1fc8387740 1 client.26581 shutdown 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.808+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 msgr2=0x5604c8775d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.808+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c6a0 tx=0x7f1fb00d2f00 comp rx=0 tx=0).stop 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 shutdown_connections 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5604c875c800 0x5604c87e6650 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604c8751f40 0x5604c8771680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fa2ff5640 0 client.26581 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604c8751620 0x5604c8771070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 --2- 172.21.15.32:0/493490182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604c874db30 0x5604c8775d00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 >> 172.21.15.32:0/493490182 conn(0x5604c8759a90 msgr2=0x5604c8753860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 shutdown_connections 2024-04-23T18:28:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 1 -- 172.21.15.32:0/493490182 wait complete. 2024-04-23T18:28:09.803 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.809+0000 7f1fc8387740 20 client.26581 trim_cache size 0 max 16384 2024-04-23T18:28:09.803 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:28:09.803 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:28:09.803 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:28:09.803 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 msgr2=0x7f1fa80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 0x7f1fa80a6250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fb801ffd0 tx=0x7f1fb801fdc0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 msgr2=0x7f1fc010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9f0 tx=0x7f1fb000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 shutdown_connections 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1fa80a3d90 0x7f1fa80a6250 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc01a36b0 0x7f1fc01a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc0107280 0x7f1fc010e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 --2- 172.21.15.32:0/1905649717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc01021b0 0x7f1fc010e170 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.810+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 >> 172.21.15.32:0/1905649717 conn(0x7f1fc00fda30 msgr2=0x7f1fc00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.811+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 shutdown_connections 2024-04-23T18:28:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:09.811+0000 7f1fc5672640 1 -- 172.21.15.32:0/1905649717 wait complete. 2024-04-23T18:28:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:10 smithi064 ceph-mon[33062]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 2.07k op/s 2024-04-23T18:28:11.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:10 smithi064 ceph-mon[33062]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:28:11.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:10 smithi032 ceph-mon[29221]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 2.07k op/s 2024-04-23T18:28:11.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:10 smithi032 ceph-mon[29221]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:28:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:10 smithi040 ceph-mon[32626]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 2.07k op/s 2024-04-23T18:28:11.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:10 smithi040 ceph-mon[32626]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:28:13.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:12 smithi064 ceph-mon[33062]: pgmap v410: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 2.52k op/s 2024-04-23T18:28:13.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:12 smithi032 ceph-mon[29221]: pgmap v410: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 2.52k op/s 2024-04-23T18:28:13.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:12 smithi040 ceph-mon[32626]: pgmap v410: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 2.52k op/s 2024-04-23T18:28:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 765.0 s, 2472.6 tps, lat 40.508 ms stddev 45.595 2024-04-23T18:28:15.167 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:14 smithi040 ceph-mon[32626]: pgmap v411: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-04-23T18:28:15.167 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:14 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:15.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:14 smithi064 ceph-mon[33062]: pgmap v411: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-04-23T18:28:15.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:14 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:14 smithi032 ceph-mon[29221]: pgmap v411: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-04-23T18:28:15.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:14 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:16 smithi064 ceph-mon[33062]: pgmap v412: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-04-23T18:28:17.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:16 smithi032 ceph-mon[29221]: pgmap v412: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-04-23T18:28:17.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:16 smithi040 ceph-mon[32626]: pgmap v412: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-04-23T18:28:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 770.0 s, 2173.4 tps, lat 46.201 ms stddev 79.266 2024-04-23T18:28:19.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:18 smithi064 ceph-mon[33062]: pgmap v413: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 MiB/s wr, 773 op/s 2024-04-23T18:28:19.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:18 smithi032 ceph-mon[29221]: pgmap v413: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 MiB/s wr, 773 op/s 2024-04-23T18:28:19.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:18 smithi040 ceph-mon[32626]: pgmap v413: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 MiB/s wr, 773 op/s 2024-04-23T18:28:21.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:20 smithi064 ceph-mon[33062]: pgmap v414: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 1.89k op/s 2024-04-23T18:28:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:20 smithi032 ceph-mon[29221]: pgmap v414: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 1.89k op/s 2024-04-23T18:28:21.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:20 smithi040 ceph-mon[32626]: pgmap v414: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 1.89k op/s 2024-04-23T18:28:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.926+0000 7f450339b640 1 -- 172.21.15.32:0/4070685767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc11b730 msgr2=0x7f44fc11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.926+0000 7f450339b640 1 --2- 172.21.15.32:0/4070685767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc11b730 0x7f44fc11db20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009be0 tx=0x7f44f402a6c0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 -- 172.21.15.32:0/4070685767 shutdown_connections 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 --2- 172.21.15.32:0/4070685767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc11b730 0x7f44fc11db20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 --2- 172.21.15.32:0/4070685767 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc10e7d0 0x7f44fc10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 --2- 172.21.15.32:0/4070685767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 -- 172.21.15.32:0/4070685767 >> 172.21.15.32:0/4070685767 conn(0x7f44fc06d030 msgr2=0x7f44fc06d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.927+0000 7f450339b640 1 -- 172.21.15.32:0/4070685767 shutdown_connections 2024-04-23T18:28:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.928+0000 7f450339b640 1 -- 172.21.15.32:0/4070685767 wait complete. 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.928+0000 7f450339b640 1 Processor -- start 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.928+0000 7f450339b640 1 -- start start 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 0x7f44fc1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc11b730 0x7f44fc1abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc120f90 con 0x7f44fc1106f0 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc120e10 con 0x7f44fc10e7d0 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f450339b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc121110 con 0x7f44fc11b730 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4500936640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4500936640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37450/0 (socket says 172.21.15.32:37450) 2024-04-23T18:28:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4500936640 1 -- 172.21.15.32:0/1000982080 learned_addr learned my addr 172.21.15.32:0/1000982080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501938640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc11b730 0x7f44fc1abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 0x7f44fc1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 -- 172.21.15.32:0/1000982080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc11b730 msgr2=0x7f44fc1abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc11b730 0x7f44fc1abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 -- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 msgr2=0x7f44fc1a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 -- 172.21.15.32:0/1000982080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc1ac440 con 0x7f44fc10e7d0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4500936640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.929+0000 7f4501137640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 0x7f44fc1a7450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e790 tx=0x7f44f800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.930+0000 7f44f27fc640 1 -- 172.21.15.32:0/1000982080 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f800eea0 con 0x7f44fc10e7d0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.930+0000 7f44f27fc640 1 -- 172.21.15.32:0/1000982080 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f800ccc0 con 0x7f44fc10e7d0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.930+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc1ac730 con 0x7f44fc10e7d0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.930+0000 7f44f27fc640 1 -- 172.21.15.32:0/1000982080 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8018680 con 0x7f44fc10e7d0 2024-04-23T18:28:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.930+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc1b3fc0 con 0x7f44fc10e7d0 2024-04-23T18:28:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.932+0000 7f44f27fc640 1 -- 172.21.15.32:0/1000982080 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f44f8010430 con 0x7f44fc10e7d0 2024-04-23T18:28:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 Processor -- start 2024-04-23T18:28:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 -- start start 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097da2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 0x55e097dad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 0x55e097dafbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e097d07970 con 0x55e097da2f40 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e097ca44a0 con 0x55e097dad800 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4506194740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e097d03930 con 0x55e097da2620 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f44f27fc640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 0x7f44e80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4501137640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 0x55e097dad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4501938640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 0x7f44e80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 0x55e097dafbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4501137640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 0x55e097dad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37460/0 (socket says 172.21.15.32:37460) 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4501137640 1 -- 172.21.15.32:0/1651902254 learned_addr learned my addr 172.21.15.32:0/1651902254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4501938640 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097da2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f44f27fc640 1 -- 172.21.15.32:0/1000982080 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f44f80d0970 con 0x7f44fc10e7d0 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 -- 172.21.15.32:0/1651902254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 msgr2=0x55e097da2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097da2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 -- 172.21.15.32:0/1651902254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 msgr2=0x55e097dad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 0x55e097dad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.933+0000 7f4500936640 1 -- 172.21.15.32:0/1651902254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e097a1a210 con 0x55e097dad800 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.934+0000 7f4500936640 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 0x55e097dafbf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009fd0 tx=0x7f44ec0119a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67f9e5dd6ede7d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.934+0000 7f4501938640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 0x7f44e80a61d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44f402abe0 tx=0x7f44f40092c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.941+0000 7f44e77fe640 1 -- 172.21.15.32:0/1651902254 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec005020 con 0x55e097dad800 2024-04-23T18:28:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.941+0000 7f44e77fe640 1 -- 172.21.15.32:0/1651902254 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44ec0047a0 con 0x55e097dad800 2024-04-23T18:28:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.941+0000 7f44e77fe640 1 -- 172.21.15.32:0/1651902254 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec00f8e0 con 0x55e097dad800 2024-04-23T18:28:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.952+0000 7f4506194740 1 -- 172.21.15.32:0/1651902254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 msgr2=0x55e097dafbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.952+0000 7f4506194740 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 0x55e097dafbf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009fd0 tx=0x7f44ec0119a0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.954+0000 7f4506194740 1 -- 172.21.15.32:0/1651902254 shutdown_connections 2024-04-23T18:28:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.954+0000 7f4506194740 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097dad800 0x55e097dafbf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.954+0000 7f4506194740 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2f40 0x55e097dad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.954+0000 7f4506194740 1 --2- 172.21.15.32:0/1651902254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097da2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.954+0000 7f4506194740 1 -- 172.21.15.32:0/1651902254 >> 172.21.15.32:0/1651902254 conn(0x55e097d9eb30 msgr2=0x55e097d9ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.963+0000 7f4506194740 1 -- 172.21.15.32:0/1651902254 shutdown_connections 2024-04-23T18:28:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.965+0000 7f4506194740 1 -- 172.21.15.32:0/1651902254 wait complete. 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.965+0000 7f4506194740 1 -- start start 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.966+0000 7f4506194740 1 Processor -- start 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.966+0000 7f4506194740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dad170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.966+0000 7f4506194740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097daa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.966+0000 7f4506194740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097daa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4501137640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4501137640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dad170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:51026/0 (socket says 172.21.15.32:51026) 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4501137640 1 -- 172.21.15.32:0/2770032953 learned_addr learned my addr 172.21.15.32:0/2770032953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e097db0430 con 0x55e097dad800 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4501938640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097daa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e097ca20b0 con 0x55e097da2f40 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e097db05b0 con 0x55e097da2620 2024-04-23T18:28:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.967+0000 7f4506194740 10 client.0 _ll_register_callbacks cb 0x55e097c8fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.968+0000 7f4500936640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097daa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.969+0000 7f4501938640 1 -- 172.21.15.32:0/2770032953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 msgr2=0x55e097dad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.969+0000 7f4501938640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.969+0000 7f4501938640 1 -- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 msgr2=0x55e097daa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.969+0000 7f4501938640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097daa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.969+0000 7f4501938640 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e097ae6620 con 0x55e097dad800 2024-04-23T18:28:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.970+0000 7f4501938640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097daa9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f44f4034040 tx=0x7f44f402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cf42d5131fbb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.970+0000 7f44e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4036040 con 0x55e097dad800 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44ec0047a0 con 0x55e097dad800 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f44f40318e0 con 0x55e097dad800 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f44f405b050 con 0x55e097dad800 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f4506194740 10 client.17073 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.971+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e097dcd3e0 con 0x55e097dad800 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44ec0047a0 con 0x55e097dad800 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f44f403c040 con 0x55e097dad800 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 10 client.17073 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 20 client.17073 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097dced90 0x55e097dd11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 4 client.17073 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.972+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e097d15390 con 0x55e097dced90 2024-04-23T18:28:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.973+0000 7f4501137640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097dced90 0x55e097dd11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.974+0000 7f4501137640 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097dced90 0x55e097dd11f0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b1c813bcc783c5 server_cookie=321820705393a8b4 in_seq=0 out_seq=0 2024-04-23T18:28:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.975+0000 7f44e4ff9640 10 client.17073 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.977+0000 7f44e4ff9640 1 -- 172.21.15.32:0/2770032953 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e097d15390 con 0x55e097dced90 2024-04-23T18:28:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.977+0000 7f44e4ff9640 10 client.17073 handle_command_reply: tid=0 2024-04-23T18:28:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.977+0000 7f4506194740 1 client.17073 shutdown 2024-04-23T18:28:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 msgr2=0x55e097daa9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097daa9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f44f4034040 tx=0x7f44f402a6c0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 shutdown_connections 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097dced90 0x55e097dd11f0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097daa9c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097daa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 --2- 172.21.15.32:0/2770032953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.978+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 >> 172.21.15.32:0/2770032953 conn(0x55e097d9eb30 msgr2=0x55e097da48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.979+0000 7f44e4ff9640 0 client.17073 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.979+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 shutdown_connections 2024-04-23T18:28:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.979+0000 7f4506194740 1 -- 172.21.15.32:0/2770032953 wait complete. 2024-04-23T18:28:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:22.979+0000 7f4506194740 20 client.17073 trim_cache size 0 max 16384 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 Processor -- start 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 -- start start 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dd6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 0x55e097dd7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1b6970 con 0x55e097da2620 2024-04-23T18:28:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1b41e0 con 0x55e097da2f40 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4506194740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc050570 con 0x55e097dad800 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4501137640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4501938640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 0x55e097dd7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4501137640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37466/0 (socket says 172.21.15.32:37466) 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4501137640 1 -- 172.21.15.32:0/781576291 learned_addr learned my addr 172.21.15.32:0/781576291 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.004+0000 7f4500936640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dd6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 -- 172.21.15.32:0/781576291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 msgr2=0x55e097dd6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dd6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 -- 172.21.15.32:0/781576291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 msgr2=0x55e097dacca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 -- 172.21.15.32:0/781576291 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e097d9a2e0 con 0x55e097dad800 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501137640 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f4501938640 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 0x55e097dd7040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009330 tx=0x7f44f4005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=988e9a60123a27c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f44e6ffd640 1 -- 172.21.15.32:0/781576291 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4036070 con 0x55e097dad800 2024-04-23T18:28:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f44e6ffd640 1 -- 172.21.15.32:0/781576291 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f4052070 con 0x55e097dad800 2024-04-23T18:28:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.005+0000 7f44e6ffd640 1 -- 172.21.15.32:0/781576291 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4004550 con 0x55e097dad800 2024-04-23T18:28:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- 172.21.15.32:0/781576291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 msgr2=0x55e097dd7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 0x55e097dd7040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009330 tx=0x7f44f4005e30 comp rx=0 tx=0).stop 2024-04-23T18:28:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- 172.21.15.32:0/781576291 shutdown_connections 2024-04-23T18:28:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097dad800 0x55e097dd7040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dd6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 --2- 172.21.15.32:0/781576291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097da2620 0x55e097dacca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- 172.21.15.32:0/781576291 >> 172.21.15.32:0/781576291 conn(0x55e097d9eb30 msgr2=0x55e097da5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- 172.21.15.32:0/781576291 shutdown_connections 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- 172.21.15.32:0/781576291 wait complete. 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.006+0000 7f4506194740 1 -- start start 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 Processor -- start 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dc1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dc1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc117c30 con 0x55e097dad800 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1190a0 con 0x55e097da2f40 2024-04-23T18:28:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc118300 con 0x55e097da2620 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4506194740 10 client.0 _ll_register_callbacks cb 0x7f44f4003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4501938640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4500936640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dc1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4501137640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dc1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4501938640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37476/0 (socket says 172.21.15.32:37476) 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4501137640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dc1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:51058/0 (socket says 172.21.15.32:51058) 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4500936640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dc1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:38800/0 (socket says 172.21.15.32:38800) 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.007+0000 7f4501938640 1 -- 172.21.15.32:0/913644197 learned_addr learned my addr 172.21.15.32:0/913644197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 -- 172.21.15.32:0/913644197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 msgr2=0x55e097dc1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dc1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 -- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 msgr2=0x55e097dc1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dc1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 -- 172.21.15.32:0/913644197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e097e338a0 con 0x55e097dad800 2024-04-23T18:28:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4501938640 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f44f403e540 tx=0x7f44f4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07cf8f73789c2be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f44e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4036040 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f44f4004040 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f4052070 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f44f402ad40 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4506194740 10 client.17079 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e097e349f0 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.008+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f4052070 con 0x55e097dad800 2024-04-23T18:28:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.009+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f44f405e020 con 0x55e097dad800 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 10 client.17079 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 10 client.17079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 20 client.17079 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097e350d0 0x55e097e374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 4 client.17079 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e097d0f200 con 0x55e097e350d0 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4501137640 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097e350d0 0x55e097e374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f4501137640 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097e350d0 0x55e097e374f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2243c25a140ae276 server_cookie=933c3ca9e276e9b6 in_seq=0 out_seq=0 2024-04-23T18:28:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.011+0000 7f44e4ff9640 10 client.17079 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f44e4ff9640 1 -- 172.21.15.32:0/913644197 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e097d0f200 con 0x55e097e350d0 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f44e4ff9640 10 client.17079 handle_command_reply: tid=0 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 client.17079 shutdown 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 msgr2=0x55e097e33360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f44f403e540 tx=0x7f44f4005fd0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 shutdown_connections 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55e097e350d0 0x55e097e374f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e097dad800 0x55e097e33360 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e097da2f40 0x55e097dc1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 --2- 172.21.15.32:0/913644197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e097da2620 0x55e097dc1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 >> 172.21.15.32:0/913644197 conn(0x55e097d9eb30 msgr2=0x55e097da9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f44e4ff9640 0 client.17079 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 shutdown_connections 2024-04-23T18:28:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 1 -- 172.21.15.32:0/913644197 wait complete. 2024-04-23T18:28:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.024+0000 7f4506194740 20 client.17079 trim_cache size 0 max 16384 2024-04-23T18:28:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:28:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:28:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:28:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 msgr2=0x7f44e80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 0x7f44e80a61d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44f402abe0 tx=0x7f44f40092c0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 msgr2=0x7f44fc1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 0x7f44fc1a7450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e790 tx=0x7f44f800ec50 comp rx=0 tx=0).stop 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 shutdown_connections 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc11b730 0x7f44fc1abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f44e80a3d10 0x7f44e80a61d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44fc1106f0 0x7f44fc1a7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 --2- 172.21.15.32:0/1000982080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc10e7d0 0x7f44fc1a7450 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 >> 172.21.15.32:0/1000982080 conn(0x7f44fc06d030 msgr2=0x7f44fc11a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 shutdown_connections 2024-04-23T18:28:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:23.025+0000 7f450339b640 1 -- 172.21.15.32:0/1000982080 wait complete. 2024-04-23T18:28:23.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:22 smithi064 ceph-mon[33062]: pgmap v415: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 1.73k op/s 2024-04-23T18:28:23.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:22 smithi032 ceph-mon[29221]: pgmap v415: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 1.73k op/s 2024-04-23T18:28:23.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:22 smithi040 ceph-mon[32626]: pgmap v415: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 1.73k op/s 2024-04-23T18:28:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 775.0 s, 2598.4 tps, lat 38.133 ms stddev 41.702 2024-04-23T18:28:25.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:24 smithi064 ceph-mon[33062]: pgmap v416: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-23T18:28:25.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:24 smithi032 ceph-mon[29221]: pgmap v416: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-23T18:28:25.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:24 smithi040 ceph-mon[32626]: pgmap v416: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-23T18:28:27.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:26 smithi064 ceph-mon[33062]: pgmap v417: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 20 MiB/s wr, 1.85k op/s 2024-04-23T18:28:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:26 smithi032 ceph-mon[29221]: pgmap v417: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 20 MiB/s wr, 1.85k op/s 2024-04-23T18:28:27.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:26 smithi040 ceph-mon[32626]: pgmap v417: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 20 MiB/s wr, 1.85k op/s 2024-04-23T18:28:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 780.0 s, 2733.6 tps, lat 36.821 ms stddev 41.141 2024-04-23T18:28:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:28 smithi032 ceph-mon[29221]: pgmap v418: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-23T18:28:29.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:28 smithi040 ceph-mon[32626]: pgmap v418: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-23T18:28:29.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:28 smithi064 ceph-mon[33062]: pgmap v418: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-23T18:28:30.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:29 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:30.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:29 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:30.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:29 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:31.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:30 smithi032 ceph-mon[29221]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-23T18:28:31.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:30 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:30 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:31.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:30 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:28:31.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:30 smithi040 ceph-mon[32626]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-23T18:28:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:30 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:30 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:30 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:28:31.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:30 smithi064 ceph-mon[33062]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-23T18:28:31.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:30 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:31.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:30 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:31.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:30 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:28:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:32 smithi032 ceph-mon[29221]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:33.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:32 smithi040 ceph-mon[32626]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:33.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:32 smithi064 ceph-mon[33062]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:33.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 785.0 s, 2233.8 tps, lat 44.715 ms stddev 47.995 2024-04-23T18:28:35.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:34 smithi032 ceph-mon[29221]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:35.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:34 smithi040 ceph-mon[32626]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:35.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:34 smithi064 ceph-mon[33062]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-04-23T18:28:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 -- 172.21.15.32:0/3445084486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec108f60 msgr2=0x7fc4ec115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/3445084486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec108f60 0x7fc4ec115420 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009ca0 tx=0x7fc4e402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 -- 172.21.15.32:0/3445084486 shutdown_connections 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/3445084486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec108f60 0x7fc4ec115420 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/3445084486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/3445084486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec10ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.145+0000 7fc4f2e91640 1 -- 172.21.15.32:0/3445084486 >> 172.21.15.32:0/3445084486 conn(0x7fc4ec076460 msgr2=0x7fc4ec078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.146+0000 7fc4f2e91640 1 -- 172.21.15.32:0/3445084486 shutdown_connections 2024-04-23T18:28:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.146+0000 7fc4f2e91640 1 -- 172.21.15.32:0/3445084486 wait complete. 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 Processor -- start 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 -- start start 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec1a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 0x7fc4ec1a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec118850 con 0x7fc4ec1085a0 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1186d0 con 0x7fc4ec108f60 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f2e91640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1189d0 con 0x7fc4ec10e660 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec1a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 0x7fc4ec1a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.142 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec1a4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57306/0 (socket says 172.21.15.32:57306) 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.147+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/4004060839 learned_addr learned my addr 172.21.15.32:0/4004060839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4f142e640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 -- 172.21.15.32:0/4004060839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 msgr2=0x7fc4ec19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 -- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 msgr2=0x7fc4ec1a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec1a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 -- 172.21.15.32:0/4004060839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec19f2b0 con 0x7fc4ec108f60 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4f142e640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 0x7fc4ec1a4e70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9f0 tx=0x7fc4d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4e9ffb640 1 -- 172.21.15.32:0/4004060839 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800ccc0 con 0x7fc4ec108f60 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec19f5a0 con 0x7fc4ec108f60 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4e9ffb640 1 -- 172.21.15.32:0/4004060839 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d80108e0 con 0x7fc4ec108f60 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.148+0000 7fc4e9ffb640 1 -- 172.21.15.32:0/4004060839 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8018880 con 0x7fc4ec108f60 2024-04-23T18:28:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.149+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec1ab7f0 con 0x7fc4ec108f60 2024-04-23T18:28:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.151+0000 7fc4e9ffb640 1 -- 172.21.15.32:0/4004060839 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc4d8018a20 con 0x7fc4ec108f60 2024-04-23T18:28:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 Processor -- start 2024-04-23T18:28:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 -- start start 2024-04-23T18:28:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d468c620 0x5573d468ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4e9ffb640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 0x7fc4dc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d4697800 0x5573d4699bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5573d45f1970 con 0x5573d4697800 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5573d458e4a0 con 0x5573d468cf40 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5573d45ed930 con 0x5573d468c620 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4e9ffb640 1 -- 172.21.15.32:0/4004060839 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fc4d80d0bb0 con 0x7fc4ec108f60 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:33474/0 (socket says 172.21.15.32:33474) 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 -- 172.21.15.32:0/3116489213 learned_addr learned my addr 172.21.15.32:0/3116489213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f142e640 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d4697800 0x5573d4699bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f142e640 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d468c620 0x5573d468ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 0x7fc4dc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 -- 172.21.15.32:0/3116489213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d468c620 msgr2=0x5573d468ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d468c620 0x5573d468ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.152+0000 7fc4ebfff640 1 -- 172.21.15.32:0/3116489213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d4697800 msgr2=0x5573d4699bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.153+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d4697800 0x5573d4699bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.153+0000 7fc4ebfff640 1 -- 172.21.15.32:0/3116489213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d4304210 con 0x5573d468cf40 2024-04-23T18:28:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.154+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80a52b0 tx=0x7fc4d8081850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167305dc593fdfeb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.154+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 0x7fc4dc0a61d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000bca0 tx=0x7fc4e000b410 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.154+0000 7fc4d6ffd640 1 -- 172.21.15.32:0/3116489213 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8081ca0 con 0x5573d468cf40 2024-04-23T18:28:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.154+0000 7fc4d6ffd640 1 -- 172.21.15.32:0/3116489213 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d809ac10 con 0x5573d468cf40 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.154+0000 7fc4d6ffd640 1 -- 172.21.15.32:0/3116489213 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d80991e0 con 0x5573d468cf40 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3116489213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 msgr2=0x5573d46972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80a52b0 tx=0x7fc4d8081850 comp rx=0 tx=0).stop 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3116489213 shutdown_connections 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d4697800 0x5573d4699bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46972c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3116489213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d468c620 0x5573d468ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3116489213 >> 172.21.15.32:0/3116489213 conn(0x5573d4688b30 msgr2=0x5573d4688f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3116489213 shutdown_connections 2024-04-23T18:28:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3116489213 wait complete. 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.155+0000 7fc4f5bad740 1 -- start start 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 Processor -- start 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d4697190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d4694480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d4694a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5573d469a3d0 con 0x5573d468cf40 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5573d458c0b0 con 0x5573d468c620 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5573d469a550 con 0x5573d4697800 2024-04-23T18:28:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f5bad740 10 client.0 _ll_register_callbacks cb 0x5573d4579d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d4694480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d4694480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57328/0 (socket says 172.21.15.32:57328) 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4ebfff640 1 -- 172.21.15.32:0/4045406611 learned_addr learned my addr 172.21.15.32:0/4045406611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d4697190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.156+0000 7fc4f142e640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d4694a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/4045406611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 msgr2=0x5573d4694a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d4694a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/4045406611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 msgr2=0x5573d4694480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d4694480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/4045406611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d43d0620 con 0x5573d468c620 2024-04-23T18:28:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f142e640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d4694a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d4697190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0010c30 tx=0x7fc4e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2866f508dd26dab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e001e8f0 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.157+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fc4e001ea90 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4d809ac10 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc4e001ec90 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4f5bad740 10 client.26588 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5573d46a6380 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d809ac10 con 0x5573d468c620 2024-04-23T18:28:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.158+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fc4e00314b0 con 0x5573d468c620 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 10 client.26588 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 10 client.26588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 20 client.26588 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d46a77f0 0x5573d46a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 4 client.26588 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573d45ff390 con 0x5573d46a77f0 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d46a77f0 0x5573d46a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d46a77f0 0x5573d46a9be0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d4309bec35dede1 server_cookie=a61af580cf6e7037 in_seq=0 out_seq=0 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.159+0000 7fc4cbfff640 10 client.26588 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4cbfff640 1 -- 172.21.15.32:0/4045406611 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573d45ff390 con 0x5573d46a77f0 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4cbfff640 10 client.26588 handle_command_reply: tid=0 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 client.26588 shutdown 2024-04-23T18:28:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 msgr2=0x5573d4697190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d4697190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0010c30 tx=0x7fc4e0004730 comp rx=0 tx=0).stop 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 shutdown_connections 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d46a77f0 0x5573d46a9be0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d4694a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d4694480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/4045406611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d4697190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 >> 172.21.15.32:0/4045406611 conn(0x5573d4688b30 msgr2=0x5573d468e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4cbfff640 0 client.26588 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 shutdown_connections 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 1 -- 172.21.15.32:0/4045406611 wait complete. 2024-04-23T18:28:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.160+0000 7fc4f5bad740 20 client.26588 trim_cache size 0 max 16384 2024-04-23T18:28:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 Processor -- start 2024-04-23T18:28:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 -- start start 2024-04-23T18:28:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d46afa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b6be0 con 0x5573d468c620 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b4450 con 0x5573d468cf40 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b22c0 con 0x5573d4697800 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57338/0 (socket says 172.21.15.32:57338) 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.180+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/865251027 learned_addr learned my addr 172.21.15.32:0/865251027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f142e640 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d46afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/865251027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 msgr2=0x5573d46afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d46afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/865251027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 msgr2=0x5573d46af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/865251027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d46842e0 con 0x5573d468c620 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000bae0 tx=0x7fc4e000bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2b35e3b5135ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4d67fc640 1 -- 172.21.15.32:0/865251027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0041070 con 0x5573d468c620 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4d67fc640 1 -- 172.21.15.32:0/865251027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e0031420 con 0x5573d468c620 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.181+0000 7fc4d67fc640 1 -- 172.21.15.32:0/865251027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00276e0 con 0x5573d468c620 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.182+0000 7fc4f5bad740 1 -- 172.21.15.32:0/865251027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 msgr2=0x5573d46a3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.182+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000bae0 tx=0x7fc4e000bb10 comp rx=0 tx=0).stop 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 -- 172.21.15.32:0/865251027 shutdown_connections 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d46afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468cf40 0x5573d46af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/865251027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468c620 0x5573d46a3290 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 -- 172.21.15.32:0/865251027 >> 172.21.15.32:0/865251027 conn(0x5573d4688b30 msgr2=0x5573d4687cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 -- 172.21.15.32:0/865251027 shutdown_connections 2024-04-23T18:28:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.189+0000 7fc4f5bad740 1 -- 172.21.15.32:0/865251027 wait complete. 2024-04-23T18:28:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.190+0000 7fc4f5bad740 1 -- start start 2024-04-23T18:28:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.190+0000 7fc4f5bad740 1 Processor -- start 2024-04-23T18:28:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.191+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.191+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d469ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.191+0000 7fc4f5bad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d469d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.191+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f0c2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:33500/0 (socket says 172.21.15.32:33500) 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b2910 con 0x5573d468cf40 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b3d80 con 0x5573d468c620 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f5bad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b2fe0 con 0x5573d4697800 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f5bad740 10 client.0 _ll_register_callbacks cb 0x7fc4e000be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.192+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/3226135619 learned_addr learned my addr 172.21.15.32:0/3226135619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.194+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/3226135619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 msgr2=0x5573d469d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.194+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d469d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.194+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/3226135619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 msgr2=0x5573d469ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.194+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d469ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.194+0000 7fc4f0c2d640 1 -- 172.21.15.32:0/3226135619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d469d770 con 0x5573d468c620 2024-04-23T18:28:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.198+0000 7fc4f0c2d640 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc4e003f040 tx=0x7fc4e00046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4433e346c44e126f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.199+0000 7fc4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:28:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.203+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0041030 con 0x5573d468c620 2024-04-23T18:28:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.203+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fc4e0037040 con 0x5573d468c620 2024-04-23T18:28:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.203+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e0031420 con 0x5573d468c620 2024-04-23T18:28:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.209+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc4e001eab0 con 0x5573d468c620 2024-04-23T18:28:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.210+0000 7fc4f5bad740 10 client.26594 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.210+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5573d470d9d0 con 0x5573d468c620 2024-04-23T18:28:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.213+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e0031420 con 0x5573d468c620 2024-04-23T18:28:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.213+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fc4e003c8c0 con 0x5573d468c620 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 10 client.26594 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 10 client.26594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 20 client.26594 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d470e060 0x5573d47104c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 4 client.26594 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5573d45f9200 con 0x5573d470e060 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.214+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d470e060 0x5573d47104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.215+0000 7fc4cbfff640 10 client.26594 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.216+0000 7fc4ebfff640 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d470e060 0x5573d47104c0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6591ebe307e5090c server_cookie=5909364af373749b in_seq=0 out_seq=0 2024-04-23T18:28:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4cbfff640 1 -- 172.21.15.32:0/3226135619 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5573d45f9200 con 0x5573d470e060 2024-04-23T18:28:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4cbfff640 10 client.26594 handle_command_reply: tid=0 2024-04-23T18:28:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 client.26594 shutdown 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 msgr2=0x5573d469f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc4e003f040 tx=0x7fc4e00046e0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 shutdown_connections 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5573d470e060 0x5573d47104c0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5573d4697800 0x5573d469d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5573d468cf40 0x5573d469ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 --2- 172.21.15.32:0/3226135619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5573d468c620 0x5573d469f830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 >> 172.21.15.32:0/3226135619 conn(0x5573d4688b30 msgr2=0x5573d46a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4cbfff640 0 client.26594 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 shutdown_connections 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 1 -- 172.21.15.32:0/3226135619 wait complete. 2024-04-23T18:28:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.233+0000 7fc4f5bad740 20 client.26594 trim_cache size 0 max 16384 2024-04-23T18:28:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:28:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:28:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:28:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 msgr2=0x7fc4dc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 0x7fc4dc0a61d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000bca0 tx=0x7fc4e000b410 comp rx=0 tx=0).stop 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 msgr2=0x7fc4ec1a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 0x7fc4ec1a4e70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9f0 tx=0x7fc4d800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 shutdown_connections 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4ec10e660 0x7fc4ec19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec108f60 0x7fc4ec1a4e70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc4dc0a3d10 0x7fc4dc0a61d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 --2- 172.21.15.32:0/4004060839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4ec1085a0 0x7fc4ec1a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 >> 172.21.15.32:0/4004060839 conn(0x7fc4ec076460 msgr2=0x7fc4ec077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 shutdown_connections 2024-04-23T18:28:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:36.235+0000 7fc4f2e91640 1 -- 172.21.15.32:0/4004060839 wait complete. 2024-04-23T18:28:37.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:36 smithi032 ceph-mon[29221]: pgmap v422: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 850 op/s 2024-04-23T18:28:37.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:36 smithi040 ceph-mon[32626]: pgmap v422: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 850 op/s 2024-04-23T18:28:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:36 smithi064 ceph-mon[33062]: pgmap v422: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 850 op/s 2024-04-23T18:28:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 790.0 s, 2585.6 tps, lat 38.772 ms stddev 45.315 2024-04-23T18:28:39.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:38 smithi032 ceph-mon[29221]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-04-23T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:38 smithi040 ceph-mon[32626]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-04-23T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:38 smithi064 ceph-mon[33062]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-04-23T18:28:41.222 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:40 smithi064 ceph-mon[33062]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.8 MiB/s wr, 911 op/s 2024-04-23T18:28:41.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:40 smithi032 ceph-mon[29221]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.8 MiB/s wr, 911 op/s 2024-04-23T18:28:41.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:40 smithi040 ceph-mon[32626]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.8 MiB/s wr, 911 op/s 2024-04-23T18:28:43.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:42 smithi032 ceph-mon[29221]: pgmap v425: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:43.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:42 smithi040 ceph-mon[32626]: pgmap v425: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:43.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:42 smithi064 ceph-mon[33062]: pgmap v425: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:43.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 795.0 s, 2842.4 tps, lat 34.893 ms stddev 40.116 2024-04-23T18:28:45.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:44 smithi032 ceph-mon[29221]: pgmap v426: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:45.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:44 smithi040 ceph-mon[32626]: pgmap v426: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:45.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:44 smithi064 ceph-mon[33062]: pgmap v426: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-23T18:28:47.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:46 smithi032 ceph-mon[29221]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.1 MiB/s wr, 914 op/s 2024-04-23T18:28:47.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:46 smithi040 ceph-mon[32626]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.1 MiB/s wr, 914 op/s 2024-04-23T18:28:47.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:46 smithi064 ceph-mon[33062]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.1 MiB/s wr, 914 op/s 2024-04-23T18:28:48.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 800.0 s, 1956.4 tps, lat 51.321 ms stddev 55.962 2024-04-23T18:28:49.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:48 smithi032 ceph-mon[29221]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-23T18:28:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 -- 172.21.15.32:0/1669730785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074c40 msgr2=0x7f8a5410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1669730785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074c40 0x7f8a5410e2a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009900 tx=0x7f8a4802a480 comp rx=0 tx=0).stop 2024-04-23T18:28:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 -- 172.21.15.32:0/1669730785 shutdown_connections 2024-04-23T18:28:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1669730785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 0x7f8a54110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1669730785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074c40 0x7f8a5410e2a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1669730785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074320 0x7f8a54074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 -- 172.21.15.32:0/1669730785 >> 172.21.15.32:0/1669730785 conn(0x7f8a540fda30 msgr2=0x7f8a540ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 -- 172.21.15.32:0/1669730785 shutdown_connections 2024-04-23T18:28:49.346 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.352+0000 7f8a5916f640 1 -- 172.21.15.32:0/1669730785 wait complete. 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 Processor -- start 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 -- start start 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 0x7f8a541a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 0x7f8a5419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54114460 con 0x7f8a54074320 2024-04-23T18:28:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541142e0 con 0x7f8a54074c40 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.353+0000 7f8a5916f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541145e0 con 0x7f8a5410e7e0 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51702/0 (socket says 172.21.15.32:51702) 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a52d9d640 1 -- 172.21.15.32:0/1003373530 learned_addr learned my addr 172.21.15.32:0/1003373530 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 0x7f8a541a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 0x7f8a5419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 -- 172.21.15.32:0/1003373530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 msgr2=0x7f8a5419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 0x7f8a5419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 -- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 msgr2=0x7f8a541a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 -- 172.21.15.32:0/1003373530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5419afb0 con 0x7f8a54074c40 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 0x7f8a541a0bb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a4802aa70 tx=0x7f8a48003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a3bfff640 1 -- 172.21.15.32:0/1003373530 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4803f070 con 0x7f8a54074c40 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a3bfff640 1 -- 172.21.15.32:0/1003373530 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a48031b50 con 0x7f8a54074c40 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5419b240 con 0x7f8a54074c40 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a3bfff640 1 -- 172.21.15.32:0/1003373530 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48035030 con 0x7f8a54074c40 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.354+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a541a74c0 con 0x7f8a54074c40 2024-04-23T18:28:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.357+0000 7f8a3bfff640 1 -- 172.21.15.32:0/1003373530 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8a48042070 con 0x7f8a54074c40 2024-04-23T18:28:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.358+0000 7f8a5bf64740 1 Processor -- start 2024-04-23T18:28:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.358+0000 7f8a3bfff640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 0x7f8a3c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.358+0000 7f8a3bfff640 1 -- 172.21.15.32:0/1003373530 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f8a480f0d20 con 0x7f8a54074c40 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.358+0000 7f8a5bf64740 1 -- start start 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.358+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 0x7f8a3c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f97f4b970 con 0x563f97ffa010 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563f97ee84a0 con 0x563f97feee30 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563f97f47930 con 0x563f97fef750 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:53190/0 (socket says 172.21.15.32:53190) 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 -- 172.21.15.32:0/1015300795 learned_addr learned my addr 172.21.15.32:0/1015300795 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 0x7f8a3c0a6250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a54069430 tx=0x7f8a4c009500 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 -- 172.21.15.32:0/1015300795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 msgr2=0x563f97ff9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 -- 172.21.15.32:0/1015300795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 msgr2=0x563f97ffc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.359+0000 7f8a52d9d640 1 -- 172.21.15.32:0/1015300795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f97c5e210 con 0x563f97feee30 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009fd0 tx=0x7f8a44011ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e58711ae772d9c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a397fa640 1 -- 172.21.15.32:0/1015300795 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4401e070 con 0x563f97feee30 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a397fa640 1 -- 172.21.15.32:0/1015300795 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4400fba0 con 0x563f97feee30 2024-04-23T18:28:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a397fa640 1 -- 172.21.15.32:0/1015300795 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44019c50 con 0x563f97feee30 2024-04-23T18:28:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1015300795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 msgr2=0x563f97fef210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.360+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009fd0 tx=0x7f8a44011ae0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1015300795 shutdown_connections 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1015300795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f97fef210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1015300795 >> 172.21.15.32:0/1015300795 conn(0x563f97feb340 msgr2=0x563f97feb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1015300795 shutdown_connections 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1015300795 wait complete. 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- start start 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 Processor -- start 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 0x563f97ff99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97ffa010 0x563f97ff72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f97ffcc40 con 0x563f97feee30 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563f97ee60b0 con 0x563f97ffa010 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.361+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563f97ffcdc0 con 0x563f97fef750 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 0x563f97ff99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5bf64740 10 client.0 _ll_register_callbacks cb 0x563f97ed3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:54018/0 (socket says 172.21.15.32:54018) 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 -- 172.21.15.32:0/3334659372 learned_addr learned my addr 172.21.15.32:0/3334659372 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5359e640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97ffa010 0x563f97ff72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 -- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97ffa010 msgr2=0x563f97ff72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97ffa010 0x563f97ff72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 -- 172.21.15.32:0/3334659372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 msgr2=0x563f97ff99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 0x563f97ff99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 -- 172.21.15.32:0/3334659372 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f97d2a620 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 0x563f97ff99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8a480b9190 tx=0x7f8a480bcbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35050078e2b0c7a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a480d6060 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f8a480dd0d0 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.362+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4400fba0 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8a480db050 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 10 client.26626 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563f97ffcc40 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4400fba0 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f8a480c5460 con 0x563f97fef750 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 10 client.26626 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 20 client.26626 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.364+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f9800a000 0x563f9800c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.365+0000 7f8a5bf64740 4 client.26626 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.365+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f97f59390 con 0x563f9800a000 2024-04-23T18:28:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.365+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f9800a000 0x563f9800c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.365+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f9800a000 0x563f9800c480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79dadc1a37e6f466 server_cookie=dd9c6f5f1c3983ea in_seq=0 out_seq=0 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.365+0000 7f8a36ff5640 10 client.26626 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3334659372 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f97f59390 con 0x563f9800a000 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a36ff5640 10 client.26626 handle_command_reply: tid=0 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 client.26626 shutdown 2024-04-23T18:28:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 msgr2=0x563f97ff6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8a480b9190 tx=0x7f8a480bcbf0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 shutdown_connections 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f9800a000 0x563f9800c480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97ffa010 0x563f97ff72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a36ff5640 0 client.26626 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff6cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3334659372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97feee30 0x563f97ff99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 >> 172.21.15.32:0/3334659372 conn(0x563f97feb340 msgr2=0x563f97fe1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 shutdown_connections 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3334659372 wait complete. 2024-04-23T18:28:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.366+0000 7f8a5bf64740 20 client.26626 trim_cache size 0 max 16384 2024-04-23T18:28:49.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:48 smithi040 ceph-mon[32626]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.386+0000 7f8a5bf64740 1 Processor -- start 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.386+0000 7f8a5bf64740 1 -- start start 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.386+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f98005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541ae360 con 0x563f97ffa010 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541abbd0 con 0x563f97feee30 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5bf64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541a9a40 con 0x563f97fef750 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51742/0 (socket says 172.21.15.32:51742) 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 -- 172.21.15.32:0/1839679096 learned_addr learned my addr 172.21.15.32:0/1839679096 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5259c640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f98005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 -- 172.21.15.32:0/1839679096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 msgr2=0x563f97ff9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 -- 172.21.15.32:0/1839679096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 msgr2=0x563f98005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f98005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 -- 172.21.15.32:0/1839679096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f97febf10 con 0x563f97ffa010 2024-04-23T18:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f98005b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a5359e640 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c003e20 tx=0x7f8a4c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8b5a664d8dfa82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a38ff9640 1 -- 172.21.15.32:0/1839679096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004d20 con 0x563f97ffa010 2024-04-23T18:28:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.387+0000 7f8a38ff9640 1 -- 172.21.15.32:0/1839679096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c0054d0 con 0x563f97ffa010 2024-04-23T18:28:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a38ff9640 1 -- 172.21.15.32:0/1839679096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c01a880 con 0x563f97ffa010 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1839679096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 msgr2=0x563f98011d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c003e20 tx=0x7f8a4c014000 comp rx=0 tx=0).stop 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1839679096 shutdown_connections 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f98011d90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ff9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/1839679096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f98005b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1839679096 >> 172.21.15.32:0/1839679096 conn(0x563f97feb340 msgr2=0x563f97fe1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1839679096 shutdown_connections 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.388+0000 7f8a5bf64740 1 -- 172.21.15.32:0/1839679096 wait complete. 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.389+0000 7f8a5bf64740 1 -- start start 2024-04-23T18:28:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.390+0000 7f8a5bf64740 1 Processor -- start 2024-04-23T18:28:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f980024d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ffd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51744/0 (socket says 172.21.15.32:51744) 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5359e640 1 -- 172.21.15.32:0/3199907301 learned_addr learned my addr 172.21.15.32:0/3199907301 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f980024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541aa090 con 0x563f97ffa010 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541ab500 con 0x563f97feee30 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541aa760 con 0x563f97fef750 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5bf64740 10 client.0 _ll_register_callbacks cb 0x7f8a480b90a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ffd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 -- 172.21.15.32:0/3199907301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 msgr2=0x563f97ffd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ffd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 -- 172.21.15.32:0/3199907301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 msgr2=0x563f97ffde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.391+0000 7f8a52d9d640 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f97ffe650 con 0x563f97feee30 2024-04-23T18:28:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a52d9d640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f980024d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a440197b0 tx=0x7f8a4400f470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c90f51e6f87952 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4401e040 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c0054d0 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f8a4400f6f0 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8a4400f8f0 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a5bf64740 10 client.26612 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.392+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563f98070710 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c0054d0 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f8a44031030 con 0x563f97feee30 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 10 client.26612 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 10 client.26612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 20 client.26612 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f98070cc0 0x563f980730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 4 client.26612 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f97f53200 con 0x563f98070cc0 2024-04-23T18:28:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.393+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f98070cc0 0x563f980730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.394+0000 7f8a5259c640 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f98070cc0 0x563f980730e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c8324173986f97 server_cookie=6cbcc6a308388be6 in_seq=0 out_seq=0 2024-04-23T18:28:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.394+0000 7f8a36ff5640 10 client.26612 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a36ff5640 1 -- 172.21.15.32:0/3199907301 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f97f53200 con 0x563f98070cc0 2024-04-23T18:28:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a36ff5640 10 client.26612 handle_command_reply: tid=0 2024-04-23T18:28:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 client.26612 shutdown 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 msgr2=0x563f980024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f980024d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a440197b0 tx=0x7f8a4400f470 comp rx=0 tx=0).stop 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 shutdown_connections 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563f98070cc0 0x563f980730e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f97ffa010 0x563f97ffde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f97fef750 0x563f97ffd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 --2- 172.21.15.32:0/3199907301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f97feee30 0x563f980024d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 >> 172.21.15.32:0/3199907301 conn(0x563f97feb340 msgr2=0x563f98004af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a36ff5640 0 client.26612 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 shutdown_connections 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 1 -- 172.21.15.32:0/3199907301 wait complete. 2024-04-23T18:28:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.402+0000 7f8a5bf64740 20 client.26612 trim_cache size 0 max 16384 2024-04-23T18:28:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:28:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:28:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:28:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 msgr2=0x7f8a3c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 0x7f8a3c0a6250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a54069430 tx=0x7f8a4c009500 comp rx=0 tx=0).stop 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 msgr2=0x7f8a541a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 0x7f8a541a0bb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a4802aa70 tx=0x7f8a48003b90 comp rx=0 tx=0).stop 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 shutdown_connections 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a5410e7e0 0x7f8a5419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a54074c40 0x7f8a541a0bb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a3c0a3d90 0x7f8a3c0a6250 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 --2- 172.21.15.32:0/1003373530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a54074320 0x7f8a541a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 >> 172.21.15.32:0/1003373530 conn(0x7f8a540fda30 msgr2=0x7f8a540fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 shutdown_connections 2024-04-23T18:28:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:28:49.403+0000 7f8a5916f640 1 -- 172.21.15.32:0/1003373530 wait complete. 2024-04-23T18:28:49.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:48 smithi064 ceph-mon[33062]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-23T18:28:51.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:50 smithi032 ceph-mon[29221]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.0 MiB/s wr, 911 op/s 2024-04-23T18:28:51.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:50 smithi040 ceph-mon[32626]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.0 MiB/s wr, 911 op/s 2024-04-23T18:28:51.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:50 smithi064 ceph-mon[33062]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.0 MiB/s wr, 911 op/s 2024-04-23T18:28:53.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:52 smithi032 ceph-mon[29221]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:53.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:52 smithi040 ceph-mon[32626]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:53.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:52 smithi064 ceph-mon[33062]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:53.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 805.0 s, 1651.8 tps, lat 60.368 ms stddev 63.956 2024-04-23T18:28:54.628 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.4 2024-04-23T18:28:54.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:28:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.783+0000 7f3da0859640 1 -- 172.21.15.32:0/4260707979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 msgr2=0x7f3d9c109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.783+0000 7f3da0859640 1 --2- 172.21.15.32:0/4260707979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c109910 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009900 tx=0x7f3d9002a440 comp rx=0 tx=0).stop 2024-04-23T18:28:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.785+0000 7f3da0859640 1 -- 172.21.15.32:0/4260707979 shutdown_connections 2024-04-23T18:28:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.785+0000 7f3da0859640 1 --2- 172.21.15.32:0/4260707979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.785+0000 7f3da0859640 1 --2- 172.21.15.32:0/4260707979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c109910 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.785+0000 7f3da0859640 1 --2- 172.21.15.32:0/4260707979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d9c110770 0x7f3d9c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.785+0000 7f3da0859640 1 -- 172.21.15.32:0/4260707979 >> 172.21.15.32:0/4260707979 conn(0x7f3d9c06e2e0 msgr2=0x7f3d9c06e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.790+0000 7f3da0859640 1 -- 172.21.15.32:0/4260707979 shutdown_connections 2024-04-23T18:28:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.790+0000 7f3da0859640 1 -- 172.21.15.32:0/4260707979 wait complete. 2024-04-23T18:28:54.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.791+0000 7f3da0859640 1 Processor -- start 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.791+0000 7f3da0859640 1 -- start start 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c1a7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d9c110770 0x7f3d9c1a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c1abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c121200 con 0x7f3d9c109490 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c121080 con 0x7f3d9c110770 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3da0859640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c121380 con 0x7f3d9c11b730 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c1a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c1a7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51766/0 (socket says 172.21.15.32:51766) 2024-04-23T18:28:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3d9a59c640 1 -- 172.21.15.32:0/2494428346 learned_addr learned my addr 172.21.15.32:0/2494428346 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.792+0000 7f3d9ad9d640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c1abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 -- 172.21.15.32:0/2494428346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d9c110770 msgr2=0x7f3d9c1a7c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:28:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d9c110770 0x7f3d9c1a7c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 -- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 msgr2=0x7f3d9c1a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c1a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 -- 172.21.15.32:0/2494428346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c1ac6c0 con 0x7f3d9c11b730 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.793+0000 7f3d9ad9d640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c1abfc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400be20 tx=0x7f3d9400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.794+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94019070 con 0x7f3d9c11b730 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.794+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d940148f0 con 0x7f3d9c11b730 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.794+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94012880 con 0x7f3d9c11b730 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.794+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c1ac9b0 con 0x7f3d9c11b730 2024-04-23T18:28:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.794+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c110f10 con 0x7f3d9c11b730 2024-04-23T18:28:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.796+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d9c06f510 con 0x7f3d9c11b730 2024-04-23T18:28:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.798+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3d9401c070 con 0x7f3d9c11b730 2024-04-23T18:28:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.798+0000 7f3d8b7fe640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 0x7f3d840a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:54.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.799+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f3d940cff50 con 0x7f3d9c11b730 2024-04-23T18:28:54.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.799+0000 7f3d9a59c640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 0x7f3d840a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:54.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.800+0000 7f3d9a59c640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 0x7f3d840a63e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009a30 tx=0x7f3d8c009190 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:54.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:54.804+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d9409c070 con 0x7f3d9c11b730 2024-04-23T18:28:55.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.149+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d9c111230 con 0x7f3d9c11b730 2024-04-23T18:28:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.150+0000 7f3d8b7fe640 1 -- 172.21.15.32:0/2494428346 <== mon.1 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11857 (secure 0 0 0) 0x7f3d940a0370 con 0x7f3d9c11b730 2024-04-23T18:28:55.143 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:28:55.143 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:28:13.809753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25348":{"gid":25348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/3901397202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3901397202},{"type":"v1","addr":"172.21.15.64:6837","nonce":3901397202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24499,24515,24551]},"id":1}]} 2024-04-23T18:28:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 45 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 msgr2=0x7f3d840a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 0x7f3d840a63e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009a30 tx=0x7f3d8c009190 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 msgr2=0x7f3d9c1abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c1abfc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400be20 tx=0x7f3d9400bf10 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 shutdown_connections 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d9c11b730 0x7f3d9c1abfc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d9c110770 0x7f3d9c1a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f3d840a3f20 0x7f3d840a63e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 --2- 172.21.15.32:0/2494428346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c109490 0x7f3d9c1a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 >> 172.21.15.32:0/2494428346 conn(0x7f3d9c06e2e0 msgr2=0x7f3d9c119720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 shutdown_connections 2024-04-23T18:28:55.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.153+0000 7f3da0859640 1 -- 172.21.15.32:0/2494428346 wait complete. 2024-04-23T18:28:55.160 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 90.4 --await-for 42.0' 2024-04-23T18:28:55.161 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 90.4 --await-for 42.0 2024-04-23T18:28:55.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:54 smithi032 ceph-mon[29221]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.334+0000 7f58adecb640 1 -- 172.21.15.32:0/3704557850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a81031e0 msgr2=0x7f58a81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.334+0000 7f58adecb640 1 --2- 172.21.15.32:0/3704557850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a81031e0 0x7f58a81035c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5898009900 tx=0x7f589802a480 comp rx=0 tx=0).stop 2024-04-23T18:28:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 -- 172.21.15.32:0/3704557850 shutdown_connections 2024-04-23T18:28:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 --2- 172.21.15.32:0/3704557850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 0x7f58a810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 --2- 172.21.15.32:0/3704557850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8103b00 0x7f58a810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 --2- 172.21.15.32:0/3704557850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a81031e0 0x7f58a81035c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 -- 172.21.15.32:0/3704557850 >> 172.21.15.32:0/3704557850 conn(0x7f58a8077fa0 msgr2=0x7f58a80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 -- 172.21.15.32:0/3704557850 shutdown_connections 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.335+0000 7f58adecb640 1 -- 172.21.15.32:0/3704557850 wait complete. 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 Processor -- start 2024-04-23T18:28:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 -- start start 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a81031e0 0x7f58a819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 0x7f58a81a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8118810 con 0x7f58a8103b00 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8118690 con 0x7f58a81031e0 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58adecb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8118990 con 0x7f58a810c5b0 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51780/0 (socket says 172.21.15.32:51780) 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1181443115 learned_addr learned my addr 172.21.15.32:0/1181443115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.336+0000 7f58a7fff640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 0x7f58a81a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a81031e0 0x7f58a819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1181443115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 msgr2=0x7f58a81a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 0x7f58a81a3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1181443115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a81031e0 msgr2=0x7f58a819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a81031e0 0x7f58a819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1181443115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a81a3d80 con 0x7f58a8103b00 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f589c00ea60 tx=0x7f589c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a4ff9640 1 -- 172.21.15.32:0/1181443115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c00cde0 con 0x7f58a8103b00 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a81a4070 con 0x7f58a8103b00 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a4ff9640 1 -- 172.21.15.32:0/1181443115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f589c0035f0 con 0x7f58a8103b00 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58a4ff9640 1 -- 172.21.15.32:0/1181443115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c010690 con 0x7f58a8103b00 2024-04-23T18:28:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.337+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a81ab900 con 0x7f58a8103b00 2024-04-23T18:28:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.340+0000 7f58a4ff9640 1 -- 172.21.15.32:0/1181443115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f589c0037a0 con 0x7f58a8103b00 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 Processor -- start 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 -- start start 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a4ff9640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 0x7f58900a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33eb8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 0x55fe33ec2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33ec3090 0x55fe33ec5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fe33e1d370 con 0x55fe33eb89c0 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe33dba2e0 con 0x55fe33eb80a0 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fe33e19330 con 0x55fe33ec3090 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a7fff640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 0x7f58900a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 0x55fe33ec2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 0x55fe33ec2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51794/0 (socket says 172.21.15.32:51794) 2024-04-23T18:28:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33eb8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1716522547 learned_addr learned my addr 172.21.15.32:0/1716522547 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.341+0000 7f58a4ff9640 1 -- 172.21.15.32:0/1181443115 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f589c0d0a70 con 0x7f58a8103b00 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 -- 172.21.15.32:0/1716522547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33ec3090 msgr2=0x55fe33ec5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33ec3090 0x55fe33ec5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 -- 172.21.15.32:0/1716522547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 msgr2=0x55fe33ec2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 0x55fe33ec2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 -- 172.21.15.32:0/1716522547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe33bfbd80 con 0x55fe33eb80a0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33eb8480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5898005ab0 tx=0x7f5898031530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a1248e5e4992e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f58a7fff640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 0x7f58900a61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55fe33eb5550 tx=0x7f58a0009a60 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f588dffb640 1 -- 172.21.15.32:0/1716522547 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898035040 con 0x55fe33eb80a0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f588dffb640 1 -- 172.21.15.32:0/1716522547 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58980054f0 con 0x55fe33eb80a0 2024-04-23T18:28:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.342+0000 7f588dffb640 1 -- 172.21.15.32:0/1716522547 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898004d60 con 0x55fe33eb80a0 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1716522547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 msgr2=0x55fe33eb8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33eb8480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5898005ab0 tx=0x7f5898031530 comp rx=0 tx=0).stop 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1716522547 shutdown_connections 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33ec3090 0x55fe33ec5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb89c0 0x55fe33ec2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1716522547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33eb8480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1716522547 >> 172.21.15.32:0/1716522547 conn(0x55fe33eb43c0 msgr2=0x55fe33eb47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1716522547 shutdown_connections 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.343+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1716522547 wait complete. 2024-04-23T18:28:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 -- start start 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 Processor -- start 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33ec0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33ec0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fe33ec5e20 con 0x55fe33ec3090 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe33db7fb0 con 0x55fe33eb80a0 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fe33ec5fa0 con 0x55fe33eb89c0 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58b0bd9740 10 client.0 _ll_register_callbacks cb 0x55fe33db79a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58a7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33ec0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33ec0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:53264/0 (socket says 172.21.15.32:53264) 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58a77fe640 1 -- 172.21.15.32:0/3489490947 learned_addr learned my addr 172.21.15.32:0/3489490947 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.344+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33ec0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a6ffd640 1 -- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 msgr2=0x55fe33ec0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33ec0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a6ffd640 1 -- 172.21.15.32:0/3489490947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 msgr2=0x55fe33ec0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a6ffd640 1 -- 172.21.15.32:0/3489490947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe33c5c510 con 0x55fe33eb89c0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.345+0000 7f58a7fff640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.347+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33ec0560 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f589c098e90 tx=0x7f589c09c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c900285d33784 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.347+0000 7f5886ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.347+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c09ca30 con 0x55fe33eb89c0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.347+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f589c0990a0 con 0x55fe33eb89c0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.347+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58980054f0 con 0x55fe33eb89c0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f589c0a0de0 con 0x55fe33eb89c0 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 10 client.26644 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55fe33aef670 con 0x55fe33eb89c0 2024-04-23T18:28:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f589c0a56a0 con 0x55fe33eb89c0 2024-04-23T18:28:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f589c004000 con 0x55fe33eb89c0 2024-04-23T18:28:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 10 client.26644 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 10 client.26644 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:28:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 20 client.26644 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33ed3360 0x55fe33ed57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 4 client.26644 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.348+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f58980054f0 con 0x55fe33ed3360 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.349+0000 7f58a77fe640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33ed3360 0x55fe33ed57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.349+0000 7f5886ffd640 10 client.26644 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.349+0000 7f58a77fe640 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33ed3360 0x55fe33ed57e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91950d8ddb07f5ec server_cookie=63f3b1fd188798e7 in_seq=0 out_seq=0 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f5886ffd640 1 -- 172.21.15.32:0/3489490947 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f58980054f0 con 0x55fe33ed3360 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f5886ffd640 10 client.26644 handle_command_reply: tid=0 2024-04-23T18:28:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 client.26644 shutdown 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 msgr2=0x55fe33ec0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33ec0560 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f589c098e90 tx=0x7f589c09c480 comp rx=0 tx=0).stop 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 shutdown_connections 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33ed3360 0x55fe33ed57e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33ec0560 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/3489490947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb80a0 0x55fe33ec0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 >> 172.21.15.32:0/3489490947 conn(0x55fe33eb43c0 msgr2=0x55fe33eb3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f5886ffd640 0 client.26644 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 shutdown_connections 2024-04-23T18:28:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.350+0000 7f58b0bd9740 1 -- 172.21.15.32:0/3489490947 wait complete. 2024-04-23T18:28:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.351+0000 7f58b0bd9740 20 client.26644 trim_cache size 0 max 16384 2024-04-23T18:28:55.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:54 smithi040 ceph-mon[32626]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.376+0000 7f58b0bd9740 1 Processor -- start 2024-04-23T18:28:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.376+0000 7f58b0bd9740 1 -- start start 2024-04-23T18:28:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33eba800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 0x55fe33ec06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b6a40 con 0x55fe33eb80a0 2024-04-23T18:28:55.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b42b0 con 0x55fe33ec3090 2024-04-23T18:28:55.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b2120 con 0x55fe33eb89c0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58a7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 0x55fe33ec06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51814/0 (socket says 172.21.15.32:51814) 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58a77fe640 1 -- 172.21.15.32:0/32962504 learned_addr learned my addr 172.21.15.32:0/32962504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.377+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33eba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 -- 172.21.15.32:0/32962504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 msgr2=0x55fe33eba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33eba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 -- 172.21.15.32:0/32962504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 msgr2=0x55fe33ec06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 0x55fe33ec06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 -- 172.21.15.32:0/32962504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe33ed5dc0 con 0x55fe33eb80a0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a7fff640 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 0x55fe33ec06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f58a77fe640 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5898000c00 tx=0x7f58980048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549e775d8ce4f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f588d7fa640 1 -- 172.21.15.32:0/32962504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898035040 con 0x55fe33eb80a0 2024-04-23T18:28:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f588d7fa640 1 -- 172.21.15.32:0/32962504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5898004440 con 0x55fe33eb80a0 2024-04-23T18:28:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.378+0000 7f588d7fa640 1 -- 172.21.15.32:0/32962504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898052600 con 0x55fe33eb80a0 2024-04-23T18:28:55.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- 172.21.15.32:0/32962504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 msgr2=0x55fe33ec20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5898000c00 tx=0x7f58980048c0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- 172.21.15.32:0/32962504 shutdown_connections 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33ec3090 0x55fe33ec06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb89c0 0x55fe33eba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/32962504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33eb80a0 0x55fe33ec20e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- 172.21.15.32:0/32962504 >> 172.21.15.32:0/32962504 conn(0x55fe33eb43c0 msgr2=0x55fe33ebffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- 172.21.15.32:0/32962504 shutdown_connections 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- 172.21.15.32:0/32962504 wait complete. 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.379+0000 7f58b0bd9740 1 -- start start 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 Processor -- start 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb80a0 0x55fe33ecb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b2770 con 0x55fe33ec3090 2024-04-23T18:28:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b3be0 con 0x55fe33eb89c0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81b2e40 con 0x55fe33eb80a0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58b0bd9740 10 client.0 _ll_register_callbacks cb 0x7f589c09c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:53290/0 (socket says 172.21.15.32:53290) 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1979213583 learned_addr learned my addr 172.21.15.32:0/1979213583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a7fff640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb80a0 0x55fe33ecb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1979213583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb80a0 msgr2=0x55fe33ecb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb80a0 0x55fe33ecb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1979213583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 msgr2=0x55fe33ec77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.380+0000 7f58a6ffd640 1 -- 172.21.15.32:0/1979213583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe33ec7fe0 con 0x55fe33eb89c0 2024-04-23T18:28:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.385+0000 7f58a6ffd640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f589c0ce7e0 tx=0x7f589c098e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d6e8625f09a959a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.385+0000 7f5886ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:28:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.385+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c087310 con 0x55fe33eb89c0 2024-04-23T18:28:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.385+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f589c09be80 con 0x55fe33eb89c0 2024-04-23T18:28:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.390+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5898004440 con 0x55fe33eb89c0 2024-04-23T18:28:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.396+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f589c0a56a0 con 0x55fe33eb89c0 2024-04-23T18:28:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.399+0000 7f58b0bd9740 10 client.26630 fetch_fsmap learned FSMap version 45 2024-04-23T18:28:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.399+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55fe33f3a4d0 con 0x55fe33eb89c0 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.405+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f589c09ca30 con 0x55fe33eb89c0 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.405+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f589c07b320 con 0x55fe33eb89c0 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 10 client.26630 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 10 client.26630 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 20 client.26630 populate_metadata read hostname 'smithi032' 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33f3ab30 0x55fe33f3cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 4 client.26630 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.4, "await_for": 42.0}] 2024-04-23T18:28:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.408+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.4, "await_for": 42.0}) -- 0x7f5898004440 con 0x55fe33f3ab30 2024-04-23T18:28:55.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.410+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33f3ab30 0x55fe33f3cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:55.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:54 smithi064 ceph-mon[33062]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 562 op/s 2024-04-23T18:28:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.429+0000 7f58a77fe640 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33f3ab30 0x55fe33f3cf50 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c817f6901a2f4d6 server_cookie=8e543e70c04ffb8e in_seq=0 out_seq=0 2024-04-23T18:28:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.429+0000 7f5886ffd640 10 client.26630 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:55.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f5886ffd640 1 -- 172.21.15.32:0/1979213583 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f5898004440 con 0x55fe33f3ab30 2024-04-23T18:28:55.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f5886ffd640 10 client.26630 handle_command_reply: tid=0 2024-04-23T18:28:55.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 client.26630 shutdown 2024-04-23T18:28:55.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 msgr2=0x55fe33ecb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f589c0ce7e0 tx=0x7f589c098e00 comp rx=0 tx=0).stop 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 shutdown_connections 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55fe33f3ab30 0x55fe33f3cf50 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fe33ec3090 0x55fe33ec77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f5886ffd640 0 client.26630 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe33eb89c0 0x55fe33ecb700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 --2- 172.21.15.32:0/1979213583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe33eb80a0 0x55fe33ecb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 >> 172.21.15.32:0/1979213583 conn(0x55fe33eb43c0 msgr2=0x55fe33eb2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 shutdown_connections 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 1 -- 172.21.15.32:0/1979213583 wait complete. 2024-04-23T18:28:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.667+0000 7f58b0bd9740 20 client.26630 trim_cache size 0 max 16384 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 45, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 46, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "71edfb18": { 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 46, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:28:55.662 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.4, 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:28:55.663 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 msgr2=0x7f58900a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 0x7f58900a61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55fe33eb5550 tx=0x7f58a0009a60 comp rx=0 tx=0).stop 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 msgr2=0x7f58a819f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f589c00ea60 tx=0x7f589c00ef20 comp rx=0 tx=0).stop 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 shutdown_connections 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58a810c5b0 0x7f58a81a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58900a3d00 0x7f58900a61c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a8103b00 0x7f58a819f2d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 --2- 172.21.15.32:0/1181443115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a81031e0 0x7f58a819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 >> 172.21.15.32:0/1181443115 conn(0x7f58a8077fa0 msgr2=0x7f58a80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 shutdown_connections 2024-04-23T18:28:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:28:55.668+0000 7f58adecb640 1 -- 172.21.15.32:0/1181443115 wait complete. 2024-04-23T18:28:55.671 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '71edfb18', quiesce took 1.0 seconds. Will release after: 30.4 2024-04-23T18:28:56.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:55 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2494428346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:56.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:55 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2494428346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:56.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:55 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2494428346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:57.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:56 smithi032 ceph-mon[29221]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 814 op/s 2024-04-23T18:28:57.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:56 smithi040 ceph-mon[32626]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 814 op/s 2024-04-23T18:28:57.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:56 smithi064 ceph-mon[33062]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 814 op/s 2024-04-23T18:28:58.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 810.0 s, 487.7 tps, lat 69.712 ms stddev 72.311 2024-04-23T18:28:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:28:58 smithi032 ceph-mon[29221]: pgmap v433: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-23T18:28:59.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:28:58 smithi040 ceph-mon[32626]: pgmap v433: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-23T18:28:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:28:58 smithi064 ceph-mon[33062]: pgmap v433: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-23T18:29:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:00 smithi032 ceph-mon[29221]: pgmap v434: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-23T18:29:01.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:01 smithi040 ceph-mon[32626]: pgmap v434: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-23T18:29:01.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:01 smithi064 ceph-mon[33062]: pgmap v434: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-23T18:29:02.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:02 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:02.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:02 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:02.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:02 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 -- 172.21.15.32:0/3933185138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e541061a0 msgr2=0x7f7e5410dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/3933185138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e541061a0 0x7f7e5410dd70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009ca0 tx=0x7f7e4c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 -- 172.21.15.32:0/3933185138 shutdown_connections 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/3933185138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e541061a0 0x7f7e5410dd70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/3933185138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e54101ad0 0x7f7e54105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/3933185138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e54101120 0x7f7e54101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 -- 172.21.15.32:0/3933185138 >> 172.21.15.32:0/3933185138 conn(0x7f7e54077fa0 msgr2=0x7f7e540783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.509+0000 7f7e5aa72640 1 -- 172.21.15.32:0/3933185138 shutdown_connections 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 -- 172.21.15.32:0/3933185138 wait complete. 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 Processor -- start 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 -- start start 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e541061a0 0x7f7e54198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 0x7f7e5419d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e54112220 con 0x7f7e54101120 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541120a0 con 0x7f7e5419cfe0 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.510+0000 7f7e5aa72640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541123a0 con 0x7f7e541061a0 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35888/0 (socket says 172.21.15.32:35888) 2024-04-23T18:29:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 -- 172.21.15.32:0/1016813965 learned_addr learned my addr 172.21.15.32:0/1016813965 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e53fff640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e541061a0 0x7f7e54198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5900f640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 0x7f7e5419d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 -- 172.21.15.32:0/1016813965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e541061a0 msgr2=0x7f7e54198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e541061a0 0x7f7e54198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 -- 172.21.15.32:0/1016813965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 msgr2=0x7f7e5419d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 0x7f7e5419d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 -- 172.21.15.32:0/1016813965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5419dad0 con 0x7f7e54101120 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5900f640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 0x7f7e5419d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800d950 tx=0x7f7e4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e51ffb640 1 -- 172.21.15.32:0/1016813965 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48017030 con 0x7f7e54101120 2024-04-23T18:29:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e51ffb640 1 -- 172.21.15.32:0/1016813965 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e48004020 con 0x7f7e54101120 2024-04-23T18:29:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5419ddc0 con 0x7f7e54101120 2024-04-23T18:29:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.511+0000 7f7e51ffb640 1 -- 172.21.15.32:0/1016813965 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480047a0 con 0x7f7e54101120 2024-04-23T18:29:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.512+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e541a5600 con 0x7f7e54101120 2024-04-23T18:29:02.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.514+0000 7f7e51ffb640 1 -- 172.21.15.32:0/1016813965 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7e48009a30 con 0x7f7e54101120 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.514+0000 7f7e5d788740 1 Processor -- start 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.514+0000 7f7e5d788740 1 -- start start 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 0x5601292a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x5601292abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560129203970 con 0x56012929e620 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e51ffb640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 0x7f7e440a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5601291a04a0 con 0x5601292a9800 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5d788740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5601291ff930 con 0x56012929ef40 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35900/0 (socket says 172.21.15.32:35900) 2024-04-23T18:29:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 -- 172.21.15.32:0/2977731036 learned_addr learned my addr 172.21.15.32:0/2977731036 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 0x7f7e440a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 0x5601292a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e53fff640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x5601292abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e51ffb640 1 -- 172.21.15.32:0/1016813965 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f7e480d01f0 con 0x7f7e54101120 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 -- 172.21.15.32:0/2977731036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 msgr2=0x5601292a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 0x5601292a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 -- 172.21.15.32:0/2977731036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 msgr2=0x5601292abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x5601292abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5900f640 1 -- 172.21.15.32:0/2977731036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560128f16210 con 0x56012929e620 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.515+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 0x5601292a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c031480 tx=0x7f7e4c0314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bde0931cd17fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5900f640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 0x7f7e440a6250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c031520 tx=0x7f7e4c042000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e3effd640 1 -- 172.21.15.32:0/2977731036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c04d070 con 0x56012929e620 2024-04-23T18:29:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e3effd640 1 -- 172.21.15.32:0/2977731036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e4c036050 con 0x56012929e620 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e3effd640 1 -- 172.21.15.32:0/2977731036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0042a0 con 0x56012929e620 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5d788740 1 -- 172.21.15.32:0/2977731036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 msgr2=0x56012929ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c031480 tx=0x7f7e4c0314b0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5d788740 1 -- 172.21.15.32:0/2977731036 shutdown_connections 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x5601292abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.516+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929ef40 0x5601292a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2977731036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929e620 0x56012929ea00 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 -- 172.21.15.32:0/2977731036 >> 172.21.15.32:0/2977731036 conn(0x56012929ab30 msgr2=0x56012929af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 -- 172.21.15.32:0/2977731036 shutdown_connections 2024-04-23T18:29:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 -- 172.21.15.32:0/2977731036 wait complete. 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 -- start start 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.517+0000 7f7e5d788740 1 Processor -- start 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5601292ac3d0 con 0x5601292a9800 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56012919e0b0 con 0x56012929ef40 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5601292ac550 con 0x56012929e620 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5d788740 10 client.0 _ll_register_callbacks cb 0x56012918bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35906/0 (socket says 172.21.15.32:35906) 2024-04-23T18:29:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5900f640 1 -- 172.21.15.32:0/1288735262 learned_addr learned my addr 172.21.15.32:0/1288735262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e53fff640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 -- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 msgr2=0x5601292a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 -- 172.21.15.32:0/1288735262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 msgr2=0x5601292a6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 -- 172.21.15.32:0/1288735262 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560128fe2620 con 0x56012929e620 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5900f640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e5880e640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a9220 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e480cd940 tx=0x7f7e48021c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfeddba9d22ce2ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e33fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.518+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480b4070 con 0x56012929e620 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f7e48098800 con 0x56012929e620 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e4c036050 con 0x56012929e620 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7e480056a0 con 0x56012929e620 2024-04-23T18:29:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 10 client.26662 fetch_fsmap learned FSMap version 46 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5601292b8390 con 0x56012929e620 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e4c036050 con 0x56012929e620 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8350+0+0 (secure 0 0 0) 0x7f7e480a0170 con 0x56012929e620 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 10 client.26662 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 10 client.26662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 20 client.26662 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5601292b98a0 0x5601292bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 4 client.26662 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.519+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560129211390 con 0x5601292b98a0 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.520+0000 7f7e53fff640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5601292b98a0 0x5601292bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.520+0000 7f7e53fff640 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5601292b98a0 0x5601292bbc90 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d23d32ac183ce3 server_cookie=8fd64bcb3cbd28c3 in_seq=0 out_seq=0 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.520+0000 7f7e33fff640 10 client.26662 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e33fff640 1 -- 172.21.15.32:0/1288735262 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560129211390 con 0x5601292b98a0 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e33fff640 10 client.26662 handle_command_reply: tid=0 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 client.26662 shutdown 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 msgr2=0x5601292a9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a9220 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e480cd940 tx=0x7f7e48021c20 comp rx=0 tx=0).stop 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 shutdown_connections 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5601292b98a0 0x5601292bbc90 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292a6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e33fff640 0 client.26662 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 --2- 172.21.15.32:0/1288735262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a9220 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 >> 172.21.15.32:0/1288735262 conn(0x56012929ab30 msgr2=0x5601292a0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 shutdown_connections 2024-04-23T18:29:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 1 -- 172.21.15.32:0/1288735262 wait complete. 2024-04-23T18:29:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.521+0000 7f7e5d788740 20 client.26662 trim_cache size 0 max 16384 2024-04-23T18:29:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 Processor -- start 2024-04-23T18:29:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 -- start start 2024-04-23T18:29:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292b5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541b0450 con 0x5601292a9800 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541adcf0 con 0x56012929ef40 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.536+0000 7f7e5d788740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541abb60 con 0x56012929e620 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35918/0 (socket says 172.21.15.32:35918) 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 -- 172.21.15.32:0/2339863717 learned_addr learned my addr 172.21.15.32:0/2339863717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e53fff640 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 -- 172.21.15.32:0/2339863717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 msgr2=0x5601292b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 -- 172.21.15.32:0/2339863717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 msgr2=0x5601292a8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 -- 172.21.15.32:0/2339863717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601292a6e90 con 0x5601292a9800 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0099b0 tx=0x7f7e4c032db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ff43dabb85762e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e3e7fc640 1 -- 172.21.15.32:0/2339863717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c04d040 con 0x5601292a9800 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e3e7fc640 1 -- 172.21.15.32:0/2339863717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e4c036070 con 0x5601292a9800 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.537+0000 7f7e3e7fc640 1 -- 172.21.15.32:0/2339863717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c02f8c0 con 0x5601292a9800 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- 172.21.15.32:0/2339863717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 msgr2=0x5601292c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0099b0 tx=0x7f7e4c032db0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- 172.21.15.32:0/2339863717 shutdown_connections 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5601292a9800 0x5601292c1580 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56012929ef40 0x5601292a8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2339863717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- 172.21.15.32:0/2339863717 >> 172.21.15.32:0/2339863717 conn(0x56012929ab30 msgr2=0x560129299cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- 172.21.15.32:0/2339863717 shutdown_connections 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- 172.21.15.32:0/2339863717 wait complete. 2024-04-23T18:29:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 -- start start 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.538+0000 7f7e5d788740 1 Processor -- start 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x56012931edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541ac1b0 con 0x56012929ef40 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541ad620 con 0x5601292a9800 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541ac880 con 0x56012929e620 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5d788740 10 client.0 _ll_register_callbacks cb 0x7f7e48098430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35930/0 (socket says 172.21.15.32:35930) 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 -- 172.21.15.32:0/2716657884 learned_addr learned my addr 172.21.15.32:0/2716657884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e5900f640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x56012931edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 -- 172.21.15.32:0/2716657884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 msgr2=0x5601292a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 -- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 msgr2=0x56012931edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x56012931edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 -- 172.21.15.32:0/2716657884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56012931f330 con 0x56012929ef40 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e53fff640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7e54100a20 tx=0x7f7e40010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c89b859d928db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.539+0000 7f7e33fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40026060 con 0x56012929ef40 2024-04-23T18:29:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f7e40004440 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e4c036070 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7e40005050 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 10 client.17154 fetch_fsmap learned FSMap version 46 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560129320540 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e4c036070 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 46) ==== 8350+0+0 (secure 0 0 0) 0x7f7e40020110 con 0x56012929ef40 2024-04-23T18:29:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 10 client.17154 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 20 client.17154 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560129320d20 0x560129323140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 4 client.17154 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56012929a980 con 0x560129320d20 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.540+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560129320d20 0x560129323140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.541+0000 7f7e5880e640 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560129320d20 0x560129323140 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34f179bb5f18dc4e server_cookie=74e19f7ebdd7d7d9 in_seq=0 out_seq=0 2024-04-23T18:29:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:02.541+0000 7f7e33fff640 10 client.17154 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:03.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:03 smithi040 ceph-mon[32626]: pgmap v435: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:03.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:03 smithi064 ceph-mon[33062]: pgmap v435: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:03.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:03 smithi032 ceph-mon[29221]: pgmap v435: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e33fff640 1 -- 172.21.15.32:0/2716657884 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56012929a980 con 0x560129320d20 2024-04-23T18:29:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e33fff640 10 client.17154 handle_command_reply: tid=0 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 client.17154 shutdown 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 msgr2=0x5601292a3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7e54100a20 tx=0x7f7e40010800 comp rx=0 tx=0).stop 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 shutdown_connections 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560129320d20 0x560129323140 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601292a9800 0x56012931edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56012929ef40 0x5601292a3b00 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 --2- 172.21.15.32:0/2716657884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56012929e620 0x5601292a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e33fff640 0 client.17154 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 >> 172.21.15.32:0/2716657884 conn(0x56012929ab30 msgr2=0x56012929ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 shutdown_connections 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.817+0000 7f7e5d788740 1 -- 172.21.15.32:0/2716657884 wait complete. 2024-04-23T18:29:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5d788740 20 client.17154 trim_cache size 0 max 16384 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 msgr2=0x7f7e440a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 0x7f7e440a6250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c031520 tx=0x7f7e4c042000 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 msgr2=0x7f7e54198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800d950 tx=0x7f7e4800de10 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 shutdown_connections 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5419cfe0 0x7f7e5419d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e541061a0 0x7f7e54198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7e440a3d90 0x7f7e440a6250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 --2- 172.21.15.32:0/1016813965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e54101120 0x7f7e54198720 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.818+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 >> 172.21.15.32:0/1016813965 conn(0x7f7e54077fa0 msgr2=0x7f7e540ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.819+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 shutdown_connections 2024-04-23T18:29:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:03.819+0000 7f7e5aa72640 1 -- 172.21.15.32:0/1016813965 wait complete. 2024-04-23T18:29:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 815.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: *** Caught signal (Segmentation fault) ** 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: in thread 7fd867294700 thread_name:md_log_replay 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd878655d20] 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2: /usr/bin/ceph-mds(+0x549372) [0x55b378cc8372] 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55b378ccbb23] 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 4: (MDLog::_replay_thread()+0x617) [0x55b378d26647] 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55b378d2fd31] 2024-04-23T18:29:04.922 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fd87864b1ca] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 7: clone() 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.648+0000 7fd867294700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: in thread 7fd867294700 thread_name:md_log_replay 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd878655d20] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2: /usr/bin/ceph-mds(+0x549372) [0x55b378cc8372] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55b378ccbb23] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 4: (MDLog::_replay_thread()+0x617) [0x55b378d26647] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55b378d2fd31] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fd87864b1ca] 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 7: clone() 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:29:04.923 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 0> 2024-04-23T18:29:04.648+0000 7fd867294700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: in thread 7fd867294700 thread_name:md_log_replay 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd878655d20] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2: /usr/bin/ceph-mds(+0x549372) [0x55b378cc8372] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55b378ccbb23] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 4: (MDLog::_replay_thread()+0x617) [0x55b378d26647] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55b378d2fd31] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fd87864b1ca] 2024-04-23T18:29:04.924 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 7: clone() 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 0> 2024-04-23T18:29:04.648+0000 7fd867294700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: in thread 7fd867294700 thread_name:md_log_replay 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd878655d20] 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2: /usr/bin/ceph-mds(+0x549372) [0x55b378cc8372] 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55b378ccbb23] 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 4: (MDLog::_replay_thread()+0x617) [0x55b378d26647] 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55b378d2fd31] 2024-04-23T18:29:04.925 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fd87864b1ca] 2024-04-23T18:29:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 7: clone() 2024-04-23T18:29:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:29:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[67692]: 2024-04-23T18:29:04.926 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:04 smithi064 podman[70317]: 2024-04-23 18:29:04.712535086 +0000 UTC m=+0.022457361 container died 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:29:05.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:05 smithi040 ceph-mon[32626]: pgmap v436: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:05.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:05 smithi064 ceph-mon[33062]: pgmap v436: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:05.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:05 smithi032 ceph-mon[29221]: pgmap v436: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-04-23T18:29:05.875 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:05 smithi064 podman[70317]: 2024-04-23 18:29:05.629552765 +0000 UTC m=+0.939475041 container remove 847dbbaf192f30afff23e2ee700255e15db30e918cfb9b7a77de4788136d7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T18:29:05.876 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-23T18:29:06.170 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-23T18:29:06.171 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:05 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Consumed 2.276s CPU time. 2024-04-23T18:29:07.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:07 smithi040 ceph-mon[32626]: pgmap v437: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 11 MiB/s wr, 718 op/s 2024-04-23T18:29:07.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:07 smithi040 ceph-mon[32626]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:29:07.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:07 smithi064 ceph-mon[33062]: pgmap v437: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 11 MiB/s wr, 718 op/s 2024-04-23T18:29:07.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:07 smithi064 ceph-mon[33062]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:29:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:07 smithi032 ceph-mon[29221]: pgmap v437: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 11 MiB/s wr, 718 op/s 2024-04-23T18:29:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:07 smithi032 ceph-mon[29221]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:29:08.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:08 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:08.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:08 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:08.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:08 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:08.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 820.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:29:09.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:09 smithi040 ceph-mon[32626]: pgmap v439: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 559 op/s 2024-04-23T18:29:09.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:09 smithi064 ceph-mon[33062]: pgmap v439: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 559 op/s 2024-04-23T18:29:09.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:09 smithi032 ceph-mon[29221]: pgmap v439: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 559 op/s 2024-04-23T18:29:11.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:11 smithi040 ceph-mon[32626]: pgmap v440: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 27 MiB/s wr, 1.35k op/s 2024-04-23T18:29:11.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:11 smithi040 ceph-mon[32626]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:29:11.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:11 smithi064 ceph-mon[33062]: pgmap v440: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 27 MiB/s wr, 1.35k op/s 2024-04-23T18:29:11.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:11 smithi064 ceph-mon[33062]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:29:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:11 smithi032 ceph-mon[29221]: pgmap v440: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 27 MiB/s wr, 1.35k op/s 2024-04-23T18:29:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:11 smithi032 ceph-mon[29221]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:29:13.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:13 smithi040 ceph-mon[32626]: pgmap v442: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 34 MiB/s wr, 1.68k op/s 2024-04-23T18:29:13.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:13 smithi064 ceph-mon[33062]: pgmap v442: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 34 MiB/s wr, 1.68k op/s 2024-04-23T18:29:13.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:13 smithi032 ceph-mon[29221]: pgmap v442: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 34 MiB/s wr, 1.68k op/s 2024-04-23T18:29:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 825.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:29:15.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:15 smithi040 ceph-mon[32626]: pgmap v443: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 1.10k op/s 2024-04-23T18:29:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:15 smithi064 ceph-mon[33062]: pgmap v443: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 1.10k op/s 2024-04-23T18:29:15.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:15 smithi032 ceph-mon[29221]: pgmap v443: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 1.10k op/s 2024-04-23T18:29:16.207 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:15 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-23T18:29:16.207 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:15 smithi064 systemd[1]: Stopped Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:29:16.207 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:15 smithi064 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service: Consumed 2.276s CPU time. 2024-04-23T18:29:16.207 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:15 smithi064 systemd[1]: Starting Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:29:16.208 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 podman[70414]: 2024-04-23 18:29:16.075997897 +0000 UTC m=+0.019223693 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:16.208 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 podman[70414]: 2024-04-23 18:29:16.21564334 +0000 UTC m=+0.158869116 container create ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T18:29:16.844 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 podman[70414]: 2024-04-23 18:29:16.468280723 +0000 UTC m=+0.411506499 container init ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:29:16.844 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 podman[70414]: 2024-04-23 18:29:16.471810428 +0000 UTC m=+0.415036209 container start ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T18:29:16.844 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 ceph-mds[70432]: -- 172.21.15.64:0/3465608919 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ffd9bda40 con 0x556ffd990c00 2024-04-23T18:29:16.844 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: starting mds.c at 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 -- 172.21.15.32:0/3504916599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4250107280 msgr2=0x7f425010ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 --2- 172.21.15.32:0/3504916599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4250107280 0x7f425010ee50 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4240009ca0 tx=0x7f424002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 -- 172.21.15.32:0/3504916599 shutdown_connections 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 --2- 172.21.15.32:0/3504916599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4250107280 0x7f425010ee50 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 --2- 172.21.15.32:0/3504916599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250102b60 0x7f4250106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 --2- 172.21.15.32:0/3504916599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42501021b0 0x7f4250102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 -- 172.21.15.32:0/3504916599 >> 172.21.15.32:0/3504916599 conn(0x7f42500fda30 msgr2=0x7f42500ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 -- 172.21.15.32:0/3504916599 shutdown_connections 2024-04-23T18:29:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.922+0000 7f4258691640 1 -- 172.21.15.32:0/3504916599 wait complete. 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 Processor -- start 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 -- start start 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4250107280 0x7f4250074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 0x7f42501a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42501121f0 con 0x7f42501021b0 2024-04-23T18:29:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4250112070 con 0x7f4250074cd0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4258691640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4250112370 con 0x7f4250107280 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f425642d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f425642d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37806/0 (socket says 172.21.15.32:37806) 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f425642d640 1 -- 172.21.15.32:0/1690711971 learned_addr learned my addr 172.21.15.32:0/1690711971 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4255c2c640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4250107280 0x7f4250074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.923+0000 7f4256c2e640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 0x7f42501a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 -- 172.21.15.32:0/1690711971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4250107280 msgr2=0x7f4250074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4250107280 0x7f4250074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 -- 172.21.15.32:0/1690711971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 msgr2=0x7f42501a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 0x7f42501a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 -- 172.21.15.32:0/1690711971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42501a6fe0 con 0x7f42501021b0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f4256c2e640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 0x7f42501a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f425642d640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9f0 tx=0x7f424800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f424f7fe640 1 -- 172.21.15.32:0/1690711971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424800cd80 con 0x7f42501021b0 2024-04-23T18:29:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f424f7fe640 1 -- 172.21.15.32:0/1690711971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42480054d0 con 0x7f42501021b0 2024-04-23T18:29:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42501a72d0 con 0x7f42501021b0 2024-04-23T18:29:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f424f7fe640 1 -- 172.21.15.32:0/1690711971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4248010690 con 0x7f42501021b0 2024-04-23T18:29:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.924+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42501a77d0 con 0x7f42501021b0 2024-04-23T18:29:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f424f7fe640 1 -- 172.21.15.32:0/1690711971 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4248004e90 con 0x7f42501021b0 2024-04-23T18:29:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 Processor -- start 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 -- start start 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352dd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c8352dfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c835237970 con 0x55c8352d2620 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c8351d44a0 con 0x55c8352dd800 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.927+0000 7f425b3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c835233930 con 0x55c8352d2f40 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37816/0 (socket says 172.21.15.32:37816) 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 -- 172.21.15.32:0/3149572343 learned_addr learned my addr 172.21.15.32:0/3149572343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f424f7fe640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 0x7f42380a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f425642d640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352dd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4255c2c640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c8352dfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 -- 172.21.15.32:0/3149572343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 msgr2=0x55c8352dd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352dd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 -- 172.21.15.32:0/3149572343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 msgr2=0x55c8352dfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c8352dfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 -- 172.21.15.32:0/3149572343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c834f4a210 con 0x55c8352d2620 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4255c2c640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c8352dfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f425642d640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352dd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 0x7f42380a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f424f7fe640 1 -- 172.21.15.32:0/1690711971 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8484+0+0 (secure 0 0 0) 0x7f42480d0c90 con 0x7f42501021b0 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.928+0000 7f4256c2e640 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4240009830 tx=0x7f4240009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51be502a040a0496 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f424cff9640 1 -- 172.21.15.32:0/3149572343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240003a30 con 0x55c8352d2620 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f424cff9640 1 -- 172.21.15.32:0/3149572343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4240036040 con 0x55c8352d2620 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f424cff9640 1 -- 172.21.15.32:0/3149572343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424002f660 con 0x55c8352d2620 2024-04-23T18:29:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 -- 172.21.15.32:0/3149572343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 msgr2=0x55c8352d2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f4256c2e640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 0x7f42380a61d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f424004c000 tx=0x7f4240049000 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4240009830 tx=0x7f4240009860 comp rx=0 tx=0).stop 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 -- 172.21.15.32:0/3149572343 shutdown_connections 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c8352dfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352dd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 --2- 172.21.15.32:0/3149572343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352d2a00 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 -- 172.21.15.32:0/3149572343 >> 172.21.15.32:0/3149572343 conn(0x55c8352ceb30 msgr2=0x55c8352cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 -- 172.21.15.32:0/3149572343 shutdown_connections 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.929+0000 7f425b3ae740 1 -- 172.21.15.32:0/3149572343 wait complete. 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 -- start start 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 Processor -- start 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352dd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352dd800 0x55c8352daa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c8352e02b0 con 0x55c8352d2f40 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c8351d20b0 con 0x55c8352d2620 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c8352e0430 con 0x55c8352dd800 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425b3ae740 10 client.0 _ll_register_callbacks cb 0x55c8351bfd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f4255c2c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f4255c2c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37826/0 (socket says 172.21.15.32:37826) 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f4255c2c640 1 -- 172.21.15.32:0/924719770 learned_addr learned my addr 172.21.15.32:0/924719770 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f425642d640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352dd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.930+0000 7f4256c2e640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352dd800 0x55c8352daa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 -- 172.21.15.32:0/924719770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352dd800 msgr2=0x55c8352daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352dd800 0x55c8352daa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 -- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 msgr2=0x55c8352dd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352dd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 -- 172.21.15.32:0/924719770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c835016620 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f425642d640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352dd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4255c2c640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f424400b940 tx=0x7f424400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3758732242b036f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424401b470 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8484+0+0 (secure 0 0 0) 0x7f424401b610 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4240036040 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f424401b810 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f425b3ae740 10 client.17172 fetch_fsmap learned FSMap version 47 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55c8352ebfe0 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.931+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4240036040 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f4244022020 con 0x55c8352d2f40 2024-04-23T18:29:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 10 client.17172 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 20 client.17172 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8352ed4a0 0x55c8352ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 4 client.17172 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c835245390 con 0x55c8352ed4a0 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425642d640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8352ed4a0 0x55c8352ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f4235ffb640 10 client.17172 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.932+0000 7f425642d640 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8352ed4a0 0x55c8352ef8d0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3511642bbd4a3274 server_cookie=984ea6c9c617fc2a in_seq=0 out_seq=0 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f4235ffb640 1 -- 172.21.15.32:0/924719770 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c835245390 con 0x55c8352ed4a0 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f4235ffb640 10 client.17172 handle_command_reply: tid=0 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 client.17172 shutdown 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 msgr2=0x55c8352da490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f424400b940 tx=0x7f424400be00 comp rx=0 tx=0).stop 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 shutdown_connections 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c8352ed4a0 0x55c8352ef8d0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352dd800 0x55c8352daa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2f40 0x55c8352da490 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f4235ffb640 0 client.17172 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 --2- 172.21.15.32:0/924719770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352dd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 >> 172.21.15.32:0/924719770 conn(0x55c8352ceb30 msgr2=0x55c8352d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 shutdown_connections 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.933+0000 7f425b3ae740 1 -- 172.21.15.32:0/924719770 wait complete. 2024-04-23T18:29:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.934+0000 7f425b3ae740 20 client.17172 trim_cache size 0 max 16384 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.948+0000 7f425b3ae740 1 Processor -- start 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 -- start start 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352da450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352d59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42501b2680 con 0x55c8352dd800 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42501aff20 con 0x55c8352d2620 2024-04-23T18:29:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425b3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f42501add90 con 0x55c8352d2f40 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37832/0 (socket says 172.21.15.32:37832) 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 -- 172.21.15.32:0/943407775 learned_addr learned my addr 172.21.15.32:0/943407775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4255c2c640 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352d59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f425642d640 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352da450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 -- 172.21.15.32:0/943407775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 msgr2=0x55c8352d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352d59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 -- 172.21.15.32:0/943407775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 msgr2=0x55c8352da450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352da450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 -- 172.21.15.32:0/943407775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8352d9440 con 0x55c8352dd800 2024-04-23T18:29:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.949+0000 7f4256c2e640 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4240004af0 tx=0x7f424002ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed23adb9ce40946 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f4237fff640 1 -- 172.21.15.32:0/943407775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424002f660 con 0x55c8352dd800 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f4237fff640 1 -- 172.21.15.32:0/943407775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4240036040 con 0x55c8352dd800 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f4237fff640 1 -- 172.21.15.32:0/943407775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240003e20 con 0x55c8352dd800 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- 172.21.15.32:0/943407775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 msgr2=0x55c8352dccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4240004af0 tx=0x7f424002ae60 comp rx=0 tx=0).stop 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- 172.21.15.32:0/943407775 shutdown_connections 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352dd800 0x55c8352dccd0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352d59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 --2- 172.21.15.32:0/943407775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352d2620 0x55c8352da450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- 172.21.15.32:0/943407775 >> 172.21.15.32:0/943407775 conn(0x55c8352ceb30 msgr2=0x55c8352cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- 172.21.15.32:0/943407775 shutdown_connections 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- 172.21.15.32:0/943407775 wait complete. 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.950+0000 7f425b3ae740 1 -- start start 2024-04-23T18:29:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 Processor -- start 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352e1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c835352a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42501ae3e0 con 0x55c8352d2620 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42501af850 con 0x55c8352dd800 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f42501aeab0 con 0x55c8352d2f40 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425b3ae740 10 client.0 _ll_register_callbacks cb 0x7f4244004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37842/0 (socket says 172.21.15.32:37842) 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 -- 172.21.15.32:0/26087207 learned_addr learned my addr 172.21.15.32:0/26087207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f4255c2c640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352e1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 -- 172.21.15.32:0/26087207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 msgr2=0x55c8352e1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352e1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 -- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 msgr2=0x55c835352a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c835352a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.951+0000 7f425642d640 1 -- 172.21.15.32:0/26087207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c835352f90 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425642d640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f42480ce460 tx=0x7f42480bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230639514f21aba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4248005680 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8484+0+0 (secure 0 0 0) 0x7f42480992c0 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4240036040 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f424809d350 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 10 client.17178 fetch_fsmap learned FSMap version 47 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55c835354200 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4240036040 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f42480d5040 con 0x55c8352d2620 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 10 client.17178 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:29:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 20 client.17178 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c835354900 0x55c835356d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 4 client.17178 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c83523f200 con 0x55c835354900 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.952+0000 7f4255c2c640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c835354900 0x55c835356d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.953+0000 7f4255c2c640 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c835354900 0x55c835356d60 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2e43c3d00d965a server_cookie=72b9816796d91c57 in_seq=0 out_seq=0 2024-04-23T18:29:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.953+0000 7f4235ffb640 10 client.17178 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f4235ffb640 1 -- 172.21.15.32:0/26087207 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c83523f200 con 0x55c835354900 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f4235ffb640 10 client.17178 handle_command_reply: tid=0 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 client.17178 shutdown 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 msgr2=0x55c8352e16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f42480ce460 tx=0x7f42480bd040 comp rx=0 tx=0).stop 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 shutdown_connections 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55c835354900 0x55c835356d60 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c8352dd800 0x55c835352a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f4235ffb640 0 client.17178 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c8352d2f40 0x55c8352e1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 --2- 172.21.15.32:0/26087207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8352d2620 0x55c8352e16d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.963+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 >> 172.21.15.32:0/26087207 conn(0x55c8352ceb30 msgr2=0x55c8352d3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 shutdown_connections 2024-04-23T18:29:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f425b3ae740 1 -- 172.21.15.32:0/26087207 wait complete. 2024-04-23T18:29:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f425b3ae740 20 client.17178 trim_cache size 0 max 16384 2024-04-23T18:29:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:29:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:29:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:29:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 msgr2=0x7f42380a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 0x7f42380a61d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f424004c000 tx=0x7f4240049000 comp rx=0 tx=0).stop 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 msgr2=0x7f4250074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9f0 tx=0x7f424800edf0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 shutdown_connections 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250074cd0 0x7f42501a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4250107280 0x7f4250074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42380a3d10 0x7f42380a61d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 --2- 172.21.15.32:0/1690711971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42501021b0 0x7f4250074250 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 >> 172.21.15.32:0/1690711971 conn(0x7f42500fda30 msgr2=0x7f42500ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 shutdown_connections 2024-04-23T18:29:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:16.964+0000 7f4258691640 1 -- 172.21.15.32:0/1690711971 wait complete. 2024-04-23T18:29:17.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:17 smithi064 ceph-mon[33062]: pgmap v444: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.00k op/s 2024-04-23T18:29:17.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:17 smithi064 ceph-mon[33062]: daemon mds.c restarted 2024-04-23T18:29:17.171 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 bash[70414]: ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 2024-04-23T18:29:17.171 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:29:16 smithi064 systemd[1]: Started Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:29:17.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:17 smithi040 ceph-mon[32626]: pgmap v444: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.00k op/s 2024-04-23T18:29:17.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:17 smithi040 ceph-mon[32626]: daemon mds.c restarted 2024-04-23T18:29:17.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:17 smithi032 ceph-mon[29221]: pgmap v444: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.00k op/s 2024-04-23T18:29:17.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:17 smithi032 ceph-mon[29221]: daemon mds.c restarted 2024-04-23T18:29:18.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:18 smithi040 ceph-mon[32626]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:29:18.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:18 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6836/1665828723,v1:172.21.15.64:6837/1665828723] up:boot 2024-04-23T18:29:18.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:18 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:29:18.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:18 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:29:18.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:18 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:18.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:18 smithi064 ceph-mon[33062]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:29:18.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:18 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6836/1665828723,v1:172.21.15.64:6837/1665828723] up:boot 2024-04-23T18:29:18.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:18 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:29:18.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:18 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:29:18.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:18 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:18.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:18 smithi032 ceph-mon[29221]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:29:18.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:18 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6836/1665828723,v1:172.21.15.64:6837/1665828723] up:boot 2024-04-23T18:29:18.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:18 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:29:18.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:18 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:29:18.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:18 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 830.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:29:19.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:19 smithi040 ceph-mon[32626]: pgmap v446: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-04-23T18:29:19.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:19 smithi032 ceph-mon[29221]: pgmap v446: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-04-23T18:29:19.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:19 smithi064 ceph-mon[33062]: pgmap v446: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-04-23T18:29:21.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:21 smithi040 ceph-mon[32626]: pgmap v447: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:29:21.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:21 smithi064 ceph-mon[33062]: pgmap v447: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:29:21.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:21 smithi032 ceph-mon[29221]: pgmap v447: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:29:23.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:23 smithi040 ceph-mon[32626]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:23.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:23 smithi064 ceph-mon[33062]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:23.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:23 smithi032 ceph-mon[29221]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 835.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:29:25.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:25 smithi040 ceph-mon[32626]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:25.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:25 smithi064 ceph-mon[33062]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:25.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:25 smithi032 ceph-mon[29221]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-04-23T18:29:26.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:29:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 -- 172.21.15.32:0/841570227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee38107270 msgr2=0x7fee3810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 --2- 172.21.15.32:0/841570227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee38107270 0x7fee3810ee40 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee30009ca0 tx=0x7fee3002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 -- 172.21.15.32:0/841570227 shutdown_connections 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 --2- 172.21.15.32:0/841570227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee38107270 0x7fee3810ee40 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 --2- 172.21.15.32:0/841570227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38102b50 0x7fee38106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 --2- 172.21.15.32:0/841570227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee381021a0 0x7fee38102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 -- 172.21.15.32:0/841570227 >> 172.21.15.32:0/841570227 conn(0x7fee380fda40 msgr2=0x7fee380ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.196+0000 7fee3f831640 1 -- 172.21.15.32:0/841570227 shutdown_connections 2024-04-23T18:29:26.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 -- 172.21.15.32:0/841570227 wait complete. 2024-04-23T18:29:26.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 Processor -- start 2024-04-23T18:29:26.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 -- start start 2024-04-23T18:29:26.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee38107270 0x7fee38074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38074cd0 0x7fee381a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fee381121e0 con 0x7fee381021a0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee38112060 con 0x7fee38107270 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.197+0000 7fee3f831640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fee38112360 con 0x7fee38074cd0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37868/0 (socket says 172.21.15.32:37868) 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 -- 172.21.15.32:0/3745330973 learned_addr learned my addr 172.21.15.32:0/3745330973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3ddce640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38074cd0 0x7fee381a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3cdcc640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee38107270 0x7fee38074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 -- 172.21.15.32:0/3745330973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38074cd0 msgr2=0x7fee381a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38074cd0 0x7fee381a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 -- 172.21.15.32:0/3745330973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee38107270 msgr2=0x7fee38074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee38107270 0x7fee38074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 -- 172.21.15.32:0/3745330973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee381a7010 con 0x7fee381021a0 2024-04-23T18:29:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3d5cd640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e9f0 tx=0x7fee2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00cd80 con 0x7fee381021a0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee2c0108e0 con 0x7fee381021a0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee381a7300 con 0x7fee381021a0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c018980 con 0x7fee381021a0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.198+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee381a7850 con 0x7fee381021a0 2024-04-23T18:29:26.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.200+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee38069250 con 0x7fee381021a0 2024-04-23T18:29:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.201+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fee2c010430 con 0x7fee381021a0 2024-04-23T18:29:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.201+0000 7fee2a7fc640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 0x7fee240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.201+0000 7fee3cdcc640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 0x7fee240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.201+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7fee2c0d0c00 con 0x7fee381021a0 2024-04-23T18:29:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.202+0000 7fee3cdcc640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 0x7fee240a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee38075ce0 tx=0x7fee34005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.206+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee2c0d7080 con 0x7fee381021a0 2024-04-23T18:29:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.432+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee3810a4a0 con 0x7fee381021a0 2024-04-23T18:29:26.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.433+0000 7fee2a7fc640 1 -- 172.21.15.32:0/3745330973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11853 (secure 0 0 0) 0x7fee2c09d2f0 con 0x7fee381021a0 2024-04-23T18:29:26.426 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:29:26.426 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:29:17.065612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:29:26.426 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 49 2024-04-23T18:29:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 msgr2=0x7fee240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 0x7fee240a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee38075ce0 tx=0x7fee34005f90 comp rx=0 tx=0).stop 2024-04-23T18:29:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 msgr2=0x7fee38074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e9f0 tx=0x7fee2c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 shutdown_connections 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee38074cd0 0x7fee381a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee38107270 0x7fee38074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fee240a3d50 0x7fee240a6210 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 --2- 172.21.15.32:0/3745330973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee381021a0 0x7fee38074250 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 >> 172.21.15.32:0/3745330973 conn(0x7fee380fda40 msgr2=0x7fee380fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 shutdown_connections 2024-04-23T18:29:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.435+0000 7fee3f831640 1 -- 172.21.15.32:0/3745330973 wait complete. 2024-04-23T18:29:26.439 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 71edfb18 --release --await' 2024-04-23T18:29:26.439 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 71edfb18 --release --await 2024-04-23T18:29:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 -- 172.21.15.32:0/1547330705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c102270 msgr2=0x7f7a8c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 --2- 172.21.15.32:0/1547330705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c102270 0x7f7a8c102650 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009930 tx=0x7f7a7c02a420 comp rx=0 tx=0).stop 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 -- 172.21.15.32:0/1547330705 shutdown_connections 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 --2- 172.21.15.32:0/1547330705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c10eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 --2- 172.21.15.32:0/1547330705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c103b50 0x7f7a8c103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 --2- 172.21.15.32:0/1547330705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c102270 0x7f7a8c102650 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 -- 172.21.15.32:0/1547330705 >> 172.21.15.32:0/1547330705 conn(0x7f7a8c0fda10 msgr2=0x7f7a8c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 -- 172.21.15.32:0/1547330705 shutdown_connections 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.556+0000 7f7a92872640 1 -- 172.21.15.32:0/1547330705 wait complete. 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 Processor -- start 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 -- start start 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 0x7f7a8c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 0x7f7a8c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c112270 con 0x7f7a8c1044f0 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1120f0 con 0x7f7a8c103b50 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a92872640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1123f0 con 0x7f7a8c102270 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a90e0f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a90e0f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37882/0 (socket says 172.21.15.32:37882) 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a90e0f640 1 -- 172.21.15.32:0/4014228386 learned_addr learned my addr 172.21.15.32:0/4014228386 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.557+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 0x7f7a8c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 0x7f7a8c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 -- 172.21.15.32:0/4014228386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 msgr2=0x7f7a8c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 0x7f7a8c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 -- 172.21.15.32:0/4014228386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 msgr2=0x7f7a8c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 0x7f7a8c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 -- 172.21.15.32:0/4014228386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c1a6cd0 con 0x7f7a8c1044f0 2024-04-23T18:29:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 0x7f7a8c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 0x7f7a8c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a90e0f640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400eaa0 tx=0x7f7a8400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a897fa640 1 -- 172.21.15.32:0/4014228386 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8400cdb0 con 0x7f7a8c1044f0 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a897fa640 1 -- 172.21.15.32:0/4014228386 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a84005020 con 0x7f7a8c1044f0 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a897fa640 1 -- 172.21.15.32:0/4014228386 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a840116d0 con 0x7f7a8c1044f0 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c1a6fc0 con 0x7f7a8c1044f0 2024-04-23T18:29:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.558+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c1a7490 con 0x7f7a8c1044f0 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 Processor -- start 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a897fa640 1 -- 172.21.15.32:0/4014228386 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7a840051d0 con 0x7f7a8c1044f0 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 -- start start 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27db0 0x563b47d31f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d32480 0x563b47d34870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563b47c8c6d0 con 0x563b47d27490 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563b47c291a0 con 0x563b47d27db0 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a95586740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563b47c88690 con 0x563b47d32480 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37892/0 (socket says 172.21.15.32:37892) 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1239367460 learned_addr learned my addr 172.21.15.32:0/1239367460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.561+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27db0 0x563b47d31f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d32480 0x563b47d34870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a897fa640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 0x7f7a740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1239367460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d32480 msgr2=0x563b47d34870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d32480 0x563b47d34870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1239367460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27db0 msgr2=0x563b47d31f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27db0 0x563b47d31f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1239367460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b47c335b0 con 0x563b47d27490 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a90e0f640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 0x7f7a740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a897fa640 1 -- 172.21.15.32:0/4014228386 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f7a840d0a00 con 0x7f7a8c1044f0 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005fd0 tx=0x7f7a80009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c54c799b2a5efa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a727fc640 1 -- 172.21.15.32:0/1239367460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a800150f0 con 0x563b47d27490 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a727fc640 1 -- 172.21.15.32:0/1239367460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a800054d0 con 0x563b47d27490 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.562+0000 7f7a727fc640 1 -- 172.21.15.32:0/1239367460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004da0 con 0x563b47d27490 2024-04-23T18:29:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- 172.21.15.32:0/1239367460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 msgr2=0x563b47d27870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a90e0f640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 0x7f7a740a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a840ce200 tx=0x7f7a84005890 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005fd0 tx=0x7f7a80009670 comp rx=0 tx=0).stop 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- 172.21.15.32:0/1239367460 shutdown_connections 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d32480 0x563b47d34870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27db0 0x563b47d31f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 --2- 172.21.15.32:0/1239367460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d27870 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- 172.21.15.32:0/1239367460 >> 172.21.15.32:0/1239367460 conn(0x563b47d237b0 msgr2=0x563b47d23b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- 172.21.15.32:0/1239367460 shutdown_connections 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- 172.21.15.32:0/1239367460 wait complete. 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 -- start start 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.563+0000 7f7a95586740 1 Processor -- start 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27490 0x563b47d31ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563b47d352b0 con 0x563b47d27db0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563b47c26b20 con 0x563b47d32480 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563b47d35430 con 0x563b47d27490 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a95586740 10 client.0 _ll_register_callbacks cb 0x563b47c26530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37904/0 (socket says 172.21.15.32:37904) 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 -- 172.21.15.32:0/232939201 learned_addr learned my addr 172.21.15.32:0/232939201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27490 0x563b47d31ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 -- 172.21.15.32:0/232939201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27490 msgr2=0x563b47d31ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27490 0x563b47d31ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 -- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 msgr2=0x563b47d2f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 -- 172.21.15.32:0/232939201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b4799e920 con 0x563b47d27db0 2024-04-23T18:29:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009930 tx=0x7f7a7c0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c3d2eab38aa9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.564+0000 7f7a6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c040070 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a800054d0 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f7a7c004240 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7a7c035040 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 10 client.17196 fetch_fsmap learned FSMap version 49 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563b47a6ad30 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a800054d0 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7f7a7c004730 con 0x563b47d27db0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 10 client.17196 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 10 client.17196 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 20 client.17196 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47d425e0 0x563b47d449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 4 client.17196 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b47c4a500 con 0x563b47d425e0 2024-04-23T18:29:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.565+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47d425e0 0x563b47d449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.566+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47d425e0 0x563b47d449d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c28813215d22c291 server_cookie=bba7ac078b825e58 in_seq=0 out_seq=0 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.566+0000 7f7a6b7fe640 10 client.17196 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.566+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/232939201 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b47c4a500 con 0x563b47d425e0 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.566+0000 7f7a6b7fe640 10 client.17196 handle_command_reply: tid=0 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 client.17196 shutdown 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 msgr2=0x563b47d2f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009930 tx=0x7f7a7c0316d0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 shutdown_connections 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47d425e0 0x563b47d449d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27db0 0x563b47d2f1d0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 --2- 172.21.15.32:0/232939201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27490 0x563b47d31ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a6b7fe640 0 client.17196 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 >> 172.21.15.32:0/232939201 conn(0x563b47d237b0 msgr2=0x563b47d229d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 shutdown_connections 2024-04-23T18:29:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 1 -- 172.21.15.32:0/232939201 wait complete. 2024-04-23T18:29:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.567+0000 7f7a95586740 20 client.17196 trim_cache size 0 max 16384 2024-04-23T18:29:26.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 Processor -- start 2024-04-23T18:29:26.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 -- start start 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d2e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1b2650 con 0x563b47d27490 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1afef0 con 0x563b47d32480 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a95586740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1add60 con 0x563b47d27db0 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37910/0 (socket says 172.21.15.32:37910) 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.584+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/3772361349 learned_addr learned my addr 172.21.15.32:0/3772361349 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d2e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/3772361349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 msgr2=0x563b47d2e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d2e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/3772361349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 msgr2=0x563b47d2a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/3772361349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b47d45050 con 0x563b47d27490 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7a80004990 tx=0x7f7a80015420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bcd92ce1bc9b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a71ffb640 1 -- 172.21.15.32:0/3772361349 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a800150c0 con 0x563b47d27490 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a71ffb640 1 -- 172.21.15.32:0/3772361349 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a80015260 con 0x563b47d27490 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.585+0000 7f7a71ffb640 1 -- 172.21.15.32:0/3772361349 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002c8b0 con 0x563b47d27490 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- 172.21.15.32:0/3772361349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 msgr2=0x563b47d3e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7a80004990 tx=0x7f7a80015420 comp rx=0 tx=0).stop 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- 172.21.15.32:0/3772361349 shutdown_connections 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d32480 0x563b47d2a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d2e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- 172.21.15.32:0/3772361349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d27490 0x563b47d3e170 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- 172.21.15.32:0/3772361349 >> 172.21.15.32:0/3772361349 conn(0x563b47d237b0 msgr2=0x563b47d29470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- 172.21.15.32:0/3772361349 shutdown_connections 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- 172.21.15.32:0/3772361349 wait complete. 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 -- start start 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 Processor -- start 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27490 0x563b47d333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.586+0000 7f7a95586740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d33910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1ae3b0 con 0x563b47d32480 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1af820 con 0x563b47d27490 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1aea80 con 0x563b47d27db0 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 10 client.0 _ll_register_callbacks cb 0x7f7a7c032cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37918/0 (socket says 172.21.15.32:37918) 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1498521164 learned_addr learned my addr 172.21.15.32:0/1498521164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d33910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27490 0x563b47d333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1498521164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 msgr2=0x563b47d33910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d33910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27490 msgr2=0x563b47d333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27490 0x563b47d333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 -- 172.21.15.32:0/1498521164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b47d38cb0 con 0x563b47d32480 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a8b7fe640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005b90 tx=0x7f7a80015560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17441b211392140e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a800150b0 con 0x563b47d32480 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f7a8002c840 con 0x563b47d32480 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b47c9a0f0 con 0x563b47d32480 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.587+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7a8002cbd0 con 0x563b47d32480 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 10 client.17202 fetch_fsmap learned FSMap version 49 2024-04-23T18:29:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563b47da9010 con 0x563b47d32480 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563b47c9a0f0 con 0x563b47d32480 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7f7a8000c070 con 0x563b47d32480 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 10 client.17202 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 10 client.17202 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 20 client.17202 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47da9560 0x563b47dab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 4 client.17202 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "71edfb18", "release": true, "await": true}] 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "71edfb18", "release": true, "await": true}) -- 0x563b47c93f60 con 0x563b47da9560 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47da9560 0x563b47dab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a6b7fe640 10 client.17202 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.588+0000 7f7a8bfff640 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47da9560 0x563b47dab980 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1426440fe7e432f2 server_cookie=12bc17e82a9af5ee in_seq=0 out_seq=0 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a6b7fe640 1 -- 172.21.15.32:0/1498521164 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x563b47c93f60 con 0x563b47da9560 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a6b7fe640 10 client.17202 handle_command_reply: tid=0 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 client.17202 shutdown 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 msgr2=0x563b47d33e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005b90 tx=0x7f7a80015560 comp rx=0 tx=0).stop 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 shutdown_connections 2024-04-23T18:29:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x563b47da9560 0x563b47dab980 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b47d32480 0x563b47d33e50 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a6b7fe640 0 client.17202 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b47d27db0 0x563b47d33910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 --2- 172.21.15.32:0/1498521164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563b47d27490 0x563b47d333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 >> 172.21.15.32:0/1498521164 conn(0x563b47d237b0 msgr2=0x563b47d3d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 shutdown_connections 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 1 -- 172.21.15.32:0/1498521164 wait complete. 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.595+0000 7f7a95586740 20 client.17202 trim_cache size 0 max 16384 2024-04-23T18:29:26.589 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 49, 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 48, 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "71edfb18": { 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 48, 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:29:26.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 90.4, 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:29:26.591 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:29:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 msgr2=0x7f7a740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 0x7f7a740a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a840ce200 tx=0x7f7a84005890 comp rx=0 tx=0).stop 2024-04-23T18:29:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 msgr2=0x7f7a8c074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400eaa0 tx=0x7f7a8400ef60 comp rx=0 tx=0).stop 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 shutdown_connections 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f7a740a3d20 0x7f7a740a61e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c1044f0 0x7f7a8c074cd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8c103b50 0x7f7a8c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 --2- 172.21.15.32:0/4014228386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a8c102270 0x7f7a8c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 >> 172.21.15.32:0/4014228386 conn(0x7f7a8c0fda10 msgr2=0x7f7a8c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 shutdown_connections 2024-04-23T18:29:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:29:26.596+0000 7f7a92872640 1 -- 172.21.15.32:0/4014228386 wait complete. 2024-04-23T18:29:26.604 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '71edfb18', seconds elapsed: 32.0 2024-04-23T18:29:27.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:27 smithi040 ceph-mon[32626]: pgmap v450: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:27.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:27 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3745330973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:27.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:27 smithi064 ceph-mon[33062]: pgmap v450: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:27.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:27 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3745330973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:27.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:27 smithi032 ceph-mon[29221]: pgmap v450: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:27.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:27 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3745330973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 840.0 s, 585.0 tps, lat 1139.607 ms stddev 5659.094 2024-04-23T18:29:29.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:29 smithi040 ceph-mon[32626]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:29.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:29 smithi064 ceph-mon[33062]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:29.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:29 smithi032 ceph-mon[29221]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3774821725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 msgr2=0x7f4ca0108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3774821725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca0108440 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009900 tx=0x7f4c9002a410 comp rx=0 tx=0).stop 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3774821725 shutdown_connections 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3774821725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006aae0 0x7f4ca010ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3774821725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006a140 0x7f4ca006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3774821725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca0108440 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3774821725 >> 172.21.15.32:0/3774821725 conn(0x7f4ca0077d30 msgr2=0x7f4ca0078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3774821725 shutdown_connections 2024-04-23T18:29:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.094+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3774821725 wait complete. 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 Processor -- start 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 -- start start 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006a140 0x7f4ca0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006aae0 0x7f4ca0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01121c0 con 0x7f4ca0108060 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0112040 con 0x7f4ca006a140 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca6acf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0112340 con 0x7f4ca006aae0 2024-04-23T18:29:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca506c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca506c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56770/0 (socket says 172.21.15.32:56770) 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca506c640 1 -- 172.21.15.32:0/3035159514 learned_addr learned my addr 172.21.15.32:0/3035159514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006a140 0x7f4ca0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.095+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006aae0 0x7f4ca0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 -- 172.21.15.32:0/3035159514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006aae0 msgr2=0x7f4ca0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006aae0 0x7f4ca0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 -- 172.21.15.32:0/3035159514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006a140 msgr2=0x7f4ca0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006a140 0x7f4ca0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 -- 172.21.15.32:0/3035159514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca019d6e0 con 0x7f4ca0108060 2024-04-23T18:29:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.096+0000 7f4ca506c640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800eaa0 tx=0x7f4c9800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.100+0000 7f4c9dffb640 1 -- 172.21.15.32:0/3035159514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800cdb0 con 0x7f4ca0108060 2024-04-23T18:29:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.101+0000 7f4c9dffb640 1 -- 172.21.15.32:0/3035159514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c98005020 con 0x7f4ca0108060 2024-04-23T18:29:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.101+0000 7f4c9dffb640 1 -- 172.21.15.32:0/3035159514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c980116d0 con 0x7f4ca0108060 2024-04-23T18:29:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.101+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca019d9d0 con 0x7f4ca0108060 2024-04-23T18:29:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.102+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca01010c0 con 0x7f4ca0108060 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.104+0000 7f4c9dffb640 1 -- 172.21.15.32:0/3035159514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4c980051d0 con 0x7f4ca0108060 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 Processor -- start 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 -- start start 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e5ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e6bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556727dc3970 con 0x556727e5ef40 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556727d604a0 con 0x556727e69800 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.105+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556727dbf930 con 0x556727e5e620 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4c9ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e6bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56784/0 (socket says 172.21.15.32:56784) 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4c9ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e6bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:59192/0 (socket says 172.21.15.32:59192) 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 -- 172.21.15.32:0/50933094 learned_addr learned my addr 172.21.15.32:0/50933094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4c9dffb640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 0x7f4c880a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e5ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca506c640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 0x7f4c880a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 -- 172.21.15.32:0/50933094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 msgr2=0x556727e5ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e5ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 -- 172.21.15.32:0/50933094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 msgr2=0x556727e6bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e6bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4ca486b640 1 -- 172.21.15.32:0/50933094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556727ad6210 con 0x556727e5ef40 2024-04-23T18:29:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.106+0000 7f4c9dffb640 1 -- 172.21.15.32:0/3035159514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f4c980d0450 con 0x7f4ca0108060 2024-04-23T18:29:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.107+0000 7f4ca486b640 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009900 tx=0x7f4c90005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb1822e7eedcf1a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.107+0000 7f4c86ffd640 1 -- 172.21.15.32:0/50933094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90035050 con 0x556727e5ef40 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.107+0000 7f4c86ffd640 1 -- 172.21.15.32:0/50933094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90004030 con 0x556727e5ef40 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.107+0000 7f4c86ffd640 1 -- 172.21.15.32:0/50933094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c900054e0 con 0x556727e5ef40 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- 172.21.15.32:0/50933094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 msgr2=0x556727e692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca506c640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 0x7f4c880a6320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c9806f050 tx=0x7f4c980c74f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009900 tx=0x7f4c90005e80 comp rx=0 tx=0).stop 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- 172.21.15.32:0/50933094 shutdown_connections 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e6bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e692c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/50933094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e5ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- 172.21.15.32:0/50933094 >> 172.21.15.32:0/50933094 conn(0x556727e5ab30 msgr2=0x556727e5af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- 172.21.15.32:0/50933094 shutdown_connections 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- 172.21.15.32:0/50933094 wait complete. 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 -- start start 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.108+0000 7f4ca97e8740 1 Processor -- start 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e668f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556727e6c420 con 0x556727e5ef40 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556727d5e0b0 con 0x556727e69800 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556727e6c5a0 con 0x556727e5e620 2024-04-23T18:29:30.105 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca97e8740 10 client.0 _ll_register_callbacks cb 0x556727d4bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56786/0 (socket says 172.21.15.32:56786) 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 -- 172.21.15.32:0/3084741725 learned_addr learned my addr 172.21.15.32:0/3084741725 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 -- 172.21.15.32:0/3084741725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 msgr2=0x556727e690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 -- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 msgr2=0x556727e668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.109+0000 7f4ca486b640 1 -- 172.21.15.32:0/3084741725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556727ba2620 con 0x556727e5ef40 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4ca486b640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4c90032f00 tx=0x7f4c9002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8341cb219bf92e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c900314a0 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f4c90031640 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c90004030 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4c9005a030 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4ca97e8740 10 client.17214 fetch_fsmap learned FSMap version 49 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.110+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556727e78320 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90004030 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7f4c900047e0 con 0x556727e5ef40 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 10 client.17214 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 10 client.17214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 20 client.17214 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727e79670 0x556727e7ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 4 client.17214 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.111+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556727dd1390 con 0x556727e79670 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.112+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727e79670 0x556727e7ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.112+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727e79670 0x556727e7ba60 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48156ce2cceab182 server_cookie=16dbee38a67a6094 in_seq=0 out_seq=0 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.113+0000 7f4c7ffff640 10 client.17214 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.113+0000 7f4c7ffff640 1 -- 172.21.15.32:0/3084741725 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556727dd1390 con 0x556727e79670 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.113+0000 7f4c7ffff640 10 client.17214 handle_command_reply: tid=0 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.114+0000 7f4ca97e8740 1 client.17214 shutdown 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.114+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 msgr2=0x556727e663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.114+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4c90032f00 tx=0x7f4c9002a410 comp rx=0 tx=0).stop 2024-04-23T18:29:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 shutdown_connections 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727e79670 0x556727e7ba60 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5ef40 0x556727e663b0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/3084741725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5e620 0x556727e690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 >> 172.21.15.32:0/3084741725 conn(0x556727e5ab30 msgr2=0x556727e60860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4c7ffff640 0 client.17214 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 shutdown_connections 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 1 -- 172.21.15.32:0/3084741725 wait complete. 2024-04-23T18:29:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.115+0000 7f4ca97e8740 20 client.17214 trim_cache size 0 max 16384 2024-04-23T18:29:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.139+0000 7f4ca97e8740 1 Processor -- start 2024-04-23T18:29:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.139+0000 7f4ca97e8740 1 -- start start 2024-04-23T18:29:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e61af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e68bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ae0e0 con 0x556727e5e620 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ab980 con 0x556727e69800 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01a97f0 con 0x556727e5ef40 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56788/0 (socket says 172.21.15.32:56788) 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 -- 172.21.15.32:0/491620769 learned_addr learned my addr 172.21.15.32:0/491620769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4ca486b640 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e61af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e68bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 -- 172.21.15.32:0/491620769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 msgr2=0x556727e61af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e61af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 -- 172.21.15.32:0/491620769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 msgr2=0x556727e68bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e68bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 -- 172.21.15.32:0/491620769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556727e66c50 con 0x556727e5e620 2024-04-23T18:29:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.140+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4c94000c00 tx=0x7f4c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d764bbc8cf608b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.141+0000 7f4c867fc640 1 -- 172.21.15.32:0/491620769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94020350 con 0x556727e5e620 2024-04-23T18:29:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.141+0000 7f4c867fc640 1 -- 172.21.15.32:0/491620769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c94005020 con 0x556727e5e620 2024-04-23T18:29:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.141+0000 7f4c867fc640 1 -- 172.21.15.32:0/491620769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c940047a0 con 0x556727e5e620 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.141+0000 7f4ca97e8740 1 -- 172.21.15.32:0/491620769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 msgr2=0x556727e75310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.141+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4c94000c00 tx=0x7f4c94004730 comp rx=0 tx=0).stop 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- 172.21.15.32:0/491620769 shutdown_connections 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e69800 0x556727e68bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e61af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/491620769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e5e620 0x556727e75310 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- 172.21.15.32:0/491620769 >> 172.21.15.32:0/491620769 conn(0x556727e5ab30 msgr2=0x556727e5ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- 172.21.15.32:0/491620769 shutdown_connections 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- 172.21.15.32:0/491620769 wait complete. 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- start start 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 Processor -- start 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e5e620 0x556727e6ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e69800 0x556727ed78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01a9e40 con 0x556727e69800 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ab2b0 con 0x556727e5e620 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.142+0000 7f4ca97e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01aa510 con 0x556727e5ef40 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca97e8740 10 client.0 _ll_register_callbacks cb 0x7f4c900044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:43202/0 (socket says 172.21.15.32:43202) 2024-04-23T18:29:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 -- 172.21.15.32:0/4212902 learned_addr learned my addr 172.21.15.32:0/4212902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e5e620 0x556727e6ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e69800 0x556727ed78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 -- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e5e620 msgr2=0x556727e6ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e5e620 0x556727e6ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 -- 172.21.15.32:0/4212902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e69800 msgr2=0x556727ed78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e69800 0x556727ed78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.143+0000 7f4ca486b640 1 -- 172.21.15.32:0/4212902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556727ed7e70 con 0x556727e5ef40 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca486b640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c90001eb0 tx=0x7f4c90005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc898a7f81ed3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90035020 con 0x556727e5ef40 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f4c9005b070 con 0x556727e5ef40 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c94005020 con 0x556727e5ef40 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4c900048d0 con 0x556727e5ef40 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 10 client.26728 fetch_fsmap learned FSMap version 49 2024-04-23T18:29:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556727edf230 con 0x556727e5ef40 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c94005020 con 0x556727e5ef40 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7f4c90032230 con 0x556727e5ef40 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 10 client.26728 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 10 client.26728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.144+0000 7f4ca97e8740 20 client.26728 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727edf970 0x556727ee1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4ca97e8740 4 client.26728 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556727dcb200 con 0x556727edf970 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727edf970 0x556727ee1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4c9ffff640 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727edf970 0x556727ee1d90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bf594847ffe8332 server_cookie=50dbd0b2db574648 in_seq=0 out_seq=0 2024-04-23T18:29:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.145+0000 7f4c7ffff640 10 client.26728 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4c7ffff640 1 -- 172.21.15.32:0/4212902 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556727dcb200 con 0x556727edf970 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4c7ffff640 10 client.26728 handle_command_reply: tid=0 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 client.26728 shutdown 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 msgr2=0x556727e6cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c90001eb0 tx=0x7f4c90005720 comp rx=0 tx=0).stop 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 shutdown_connections 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556727edf970 0x556727ee1d90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556727e69800 0x556727ed78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556727e5ef40 0x556727e6cf90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4c7ffff640 0 client.26728 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 --2- 172.21.15.32:0/4212902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556727e5e620 0x556727e6ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.155+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 >> 172.21.15.32:0/4212902 conn(0x556727e5ab30 msgr2=0x556727e664d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 shutdown_connections 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca97e8740 1 -- 172.21.15.32:0/4212902 wait complete. 2024-04-23T18:29:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca97e8740 20 client.26728 trim_cache size 0 max 16384 2024-04-23T18:29:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:29:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:29:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:29:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 msgr2=0x7f4c880a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 0x7f4c880a6320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c9806f050 tx=0x7f4c980c74f0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 msgr2=0x7f4ca019cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.156+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800eaa0 tx=0x7f4c9800ef60 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 shutdown_connections 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c880a3e60 0x7f4c880a6320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca0108060 0x7f4ca019cfb0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ca006aae0 0x7f4ca0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 --2- 172.21.15.32:0/3035159514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca006a140 0x7f4ca0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 >> 172.21.15.32:0/3035159514 conn(0x7f4ca0077d30 msgr2=0x7f4ca00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 shutdown_connections 2024-04-23T18:29:30.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:30.157+0000 7f4ca6acf640 1 -- 172.21.15.32:0/3035159514 wait complete. 2024-04-23T18:29:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:31 smithi040 ceph-mon[32626]: pgmap v452: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-04-23T18:29:31.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:31 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:31.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:31 smithi064 ceph-mon[33062]: pgmap v452: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-04-23T18:29:31.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:31 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:31.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:31 smithi032 ceph-mon[29221]: pgmap v452: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-04-23T18:29:31.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:31 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:32.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:32 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:32.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:32 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:32.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:32 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:32.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:32 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:29:32.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:32 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:32.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:32 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:32.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:32 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:32.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:32 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:29:32.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:32 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:32.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:32 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:32.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:32 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:32.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:32 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:29:33.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:33 smithi040 ceph-mon[32626]: pgmap v453: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:33.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:33 smithi064 ceph-mon[33062]: pgmap v453: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:33.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:33 smithi032 ceph-mon[29221]: pgmap v453: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:33.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 845.0 s, 1810.0 tps, lat 55.298 ms stddev 61.526 2024-04-23T18:29:35.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:35 smithi064 ceph-mon[33062]: pgmap v454: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:35.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:35 smithi032 ceph-mon[29221]: pgmap v454: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:35.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:35 smithi040 ceph-mon[32626]: pgmap v454: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-04-23T18:29:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:37 smithi064 ceph-mon[33062]: pgmap v455: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 493 op/s 2024-04-23T18:29:37.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:37 smithi032 ceph-mon[29221]: pgmap v455: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 493 op/s 2024-04-23T18:29:37.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:37 smithi040 ceph-mon[32626]: pgmap v455: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 493 op/s 2024-04-23T18:29:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 850.0 s, 2239.0 tps, lat 44.776 ms stddev 52.166 2024-04-23T18:29:39.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:39 smithi064 ceph-mon[33062]: pgmap v456: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 490 op/s 2024-04-23T18:29:39.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:39 smithi032 ceph-mon[29221]: pgmap v456: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 490 op/s 2024-04-23T18:29:39.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:39 smithi040 ceph-mon[32626]: pgmap v456: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 490 op/s 2024-04-23T18:29:41.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:41 smithi064 ceph-mon[33062]: pgmap v457: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 19 MiB/s wr, 775 op/s 2024-04-23T18:29:41.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:41 smithi032 ceph-mon[29221]: pgmap v457: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 19 MiB/s wr, 775 op/s 2024-04-23T18:29:41.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:41 smithi040 ceph-mon[32626]: pgmap v457: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 19 MiB/s wr, 775 op/s 2024-04-23T18:29:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.282+0000 7f6709c74640 1 -- 172.21.15.32:0/1323666253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6704103ae0 msgr2=0x7f670410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.282+0000 7f6709c74640 1 --2- 172.21.15.32:0/1323666253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6704103ae0 0x7f670410bfc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009900 tx=0x7f66f802a480 comp rx=0 tx=0).stop 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 -- 172.21.15.32:0/1323666253 shutdown_connections 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 --2- 172.21.15.32:0/1323666253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f670410c590 0x7f670410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 --2- 172.21.15.32:0/1323666253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6704103ae0 0x7f670410bfc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 --2- 172.21.15.32:0/1323666253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67041031c0 0x7f67041035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 -- 172.21.15.32:0/1323666253 >> 172.21.15.32:0/1323666253 conn(0x7f6704077fa0 msgr2=0x7f67040783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 -- 172.21.15.32:0/1323666253 shutdown_connections 2024-04-23T18:29:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.283+0000 7f6709c74640 1 -- 172.21.15.32:0/1323666253 wait complete. 2024-04-23T18:29:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 Processor -- start 2024-04-23T18:29:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 -- start start 2024-04-23T18:29:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 0x7f6704198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6704103ae0 0x7f6704198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 0x7f670419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67041121b0 con 0x7f67041031c0 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6704112030 con 0x7f670410c590 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.284+0000 7f6709c74640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6704112330 con 0x7f6704103ae0 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f67037fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 0x7f6704198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6704103ae0 0x7f6704198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f67037fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 0x7f6704198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32924/0 (socket says 172.21.15.32:32924) 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 0x7f670419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f67037fe640 1 -- 172.21.15.32:0/2668629693 learned_addr learned my addr 172.21.15.32:0/2668629693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 -- 172.21.15.32:0/2668629693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6704103ae0 msgr2=0x7f6704198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6704103ae0 0x7f6704198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 -- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 msgr2=0x7f6704198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 0x7f6704198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.285+0000 7f6703fff640 1 -- 172.21.15.32:0/2668629693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670419d720 con 0x7f670410c590 2024-04-23T18:29:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6703fff640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 0x7f670419cff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009fd0 tx=0x7f66fc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6700ff9640 1 -- 172.21.15.32:0/2668629693 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00ce20 con 0x7f670410c590 2024-04-23T18:29:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6700ff9640 1 -- 172.21.15.32:0/2668629693 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66fc005020 con 0x7f670410c590 2024-04-23T18:29:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670419da10 con 0x7f670410c590 2024-04-23T18:29:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6700ff9640 1 -- 172.21.15.32:0/2668629693 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc010690 con 0x7f670410c590 2024-04-23T18:29:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.286+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6704100d40 con 0x7f670410c590 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.288+0000 7f6700ff9640 1 -- 172.21.15.32:0/2668629693 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f66fc0054d0 con 0x7f670410c590 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 Processor -- start 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 -- start start 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617acea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ad9800 0x560617adbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560617a33970 con 0x560617ace620 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5606179d04a0 con 0x560617ad9800 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f670c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560617a2f930 con 0x560617acef40 2024-04-23T18:29:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6700ff9640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 0x7f66ec0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617acea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f67037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f67037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:58622/0 (socket says 172.21.15.32:58622) 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617acea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32938/0 (socket says 172.21.15.32:32938) 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f67037fe640 1 -- 172.21.15.32:0/3682851823 learned_addr learned my addr 172.21.15.32:0/3682851823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6702ffd640 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ad9800 0x560617adbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6700ff9640 1 -- 172.21.15.32:0/2668629693 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f66fc0d0e00 con 0x7f670410c590 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.289+0000 7f6703fff640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 0x7f66ec0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 -- 172.21.15.32:0/3682851823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ad9800 msgr2=0x560617adbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ad9800 0x560617adbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 -- 172.21.15.32:0/3682851823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 msgr2=0x560617acea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617acea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 -- 172.21.15.32:0/3682851823 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560617746210 con 0x560617acef40 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.290+0000 7f67037fe640 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009e10 tx=0x7f66f4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b6ad63d98850c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.291+0000 7f66e9ffb640 1 -- 172.21.15.32:0/3682851823 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f401e070 con 0x560617acef40 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.291+0000 7f66e9ffb640 1 -- 172.21.15.32:0/3682851823 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f4019ae0 con 0x560617acef40 2024-04-23T18:29:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.291+0000 7f66e9ffb640 1 -- 172.21.15.32:0/3682851823 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f400fb70 con 0x560617acef40 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f6703fff640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 0x7f66ec0a6250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66fc099760 tx=0x7f66fc0057c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- 172.21.15.32:0/3682851823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 msgr2=0x560617ad92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009e10 tx=0x7f66f4011810 comp rx=0 tx=0).stop 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- 172.21.15.32:0/3682851823 shutdown_connections 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ad9800 0x560617adbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617ad92c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 --2- 172.21.15.32:0/3682851823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617acea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- 172.21.15.32:0/3682851823 >> 172.21.15.32:0/3682851823 conn(0x560617acab30 msgr2=0x560617acaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- 172.21.15.32:0/3682851823 shutdown_connections 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- 172.21.15.32:0/3682851823 wait complete. 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 -- start start 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.292+0000 7f670c981740 1 Processor -- start 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 0x560617ad6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560617adc430 con 0x560617ace620 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5606179ce0b0 con 0x560617acef40 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560617adc5b0 con 0x560617ad9800 2024-04-23T18:29:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f670c981740 10 client.0 _ll_register_callbacks cb 0x5606179bbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32946/0 (socket says 172.21.15.32:32946) 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 -- 172.21.15.32:0/482744299 learned_addr learned my addr 172.21.15.32:0/482744299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f67037fe640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 0x560617ad6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 -- 172.21.15.32:0/482744299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 msgr2=0x560617ad6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 -- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 msgr2=0x560617ad6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 0x560617ad6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f6702ffd640 1 -- 172.21.15.32:0/482744299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560617812620 con 0x560617ace620 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.293+0000 7f67037fe640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 0x560617ad6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.294+0000 7f66e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.294+0000 7f6702ffd640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f66f8005c40 tx=0x7f66f8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f529e5ddcec0d9c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.294+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f804f060 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.294+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f66f8043040 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.294+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66f4019ae0 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f66f804f060 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 10 client.17232 fetch_fsmap learned FSMap version 50 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x560617ae8380 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f4019ae0 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 50) ==== 8342+0+0 (secure 0 0 0) 0x7f66f8046030 con 0x560617ace620 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 10 client.17232 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 10 client.17232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 20 client.17232 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617aebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 4 client.17232 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560617a41390 con 0x560617ae9840 2024-04-23T18:29:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.295+0000 7f67037fe640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617aebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.296+0000 7f66e2ffd640 10 client.17232 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.296+0000 7f67037fe640 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617aebc30 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad82fb6e2e9d3ae server_cookie=238a8ddbaca4fbb4 in_seq=0 out_seq=0 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.296+0000 7f66e2ffd640 1 -- 172.21.15.32:0/482744299 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560617a41390 con 0x560617ae9840 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.296+0000 7f66e2ffd640 10 client.17232 handle_command_reply: tid=0 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.296+0000 7f670c981740 1 client.17232 shutdown 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 msgr2=0x560617ad9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f66f8005c40 tx=0x7f66f8005b50 comp rx=0 tx=0).stop 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 shutdown_connections 2024-04-23T18:29:43.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617aebc30 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f66e2ffd640 0 client.17232 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617acef40 0x560617ad6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 --2- 172.21.15.32:0/482744299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ace620 0x560617ad9190 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 >> 172.21.15.32:0/482744299 conn(0x560617acab30 msgr2=0x560617ad08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 shutdown_connections 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 1 -- 172.21.15.32:0/482744299 wait complete. 2024-04-23T18:29:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.297+0000 7f670c981740 20 client.17232 trim_cache size 0 max 16384 2024-04-23T18:29:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 Processor -- start 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 -- start start 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617ae52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67041ae1c0 con 0x560617acef40 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67041aba90 con 0x560617ace620 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f670c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f67041a98e0 con 0x560617ad9800 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f67037fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f67037fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32952/0 (socket says 172.21.15.32:32952) 2024-04-23T18:29:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f67037fe640 1 -- 172.21.15.32:0/2206877120 learned_addr learned my addr 172.21.15.32:0/2206877120 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f6702ffd640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617ae52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.317+0000 7f6702ffd640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 -- 172.21.15.32:0/2206877120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 msgr2=0x560617ad83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 -- 172.21.15.32:0/2206877120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 msgr2=0x560617ae52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617ae52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 -- 172.21.15.32:0/2206877120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560617ac62e0 con 0x560617acef40 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f6702ffd640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f67037fe640 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f66f401afd0 tx=0x7f66f4005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2fa5b347c02d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f66e97fa640 1 -- 172.21.15.32:0/2206877120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4029e60 con 0x560617acef40 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f66e97fa640 1 -- 172.21.15.32:0/2206877120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f401f040 con 0x560617acef40 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.318+0000 7f66e97fa640 1 -- 172.21.15.32:0/2206877120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f401e040 con 0x560617acef40 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 -- 172.21.15.32:0/2206877120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 msgr2=0x560617ad53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f66f401afd0 tx=0x7f66f4005280 comp rx=0 tx=0).stop 2024-04-23T18:29:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 -- 172.21.15.32:0/2206877120 shutdown_connections 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617ad9800 0x560617ad83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617acef40 0x560617ad53c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 --2- 172.21.15.32:0/2206877120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617ae52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 -- 172.21.15.32:0/2206877120 >> 172.21.15.32:0/2206877120 conn(0x560617acab30 msgr2=0x560617ad7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 -- 172.21.15.32:0/2206877120 shutdown_connections 2024-04-23T18:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.319+0000 7f670c981740 1 -- 172.21.15.32:0/2206877120 wait complete. 2024-04-23T18:29:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 -- start start 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 Processor -- start 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617adc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617adcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ad9800 0x560617b4dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67041a9f50 con 0x560617ad9800 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67041ab3c0 con 0x560617ace620 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f67041aa620 con 0x560617acef40 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f670c981740 10 client.0 _ll_register_callbacks cb 0x7f66f8032ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ad9800 0x560617b4dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f67037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617adcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617adc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f67037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617adcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:58644/0 (socket says 172.21.15.32:58644) 2024-04-23T18:29:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f67037fe640 1 -- 172.21.15.32:0/1040038832 learned_addr learned my addr 172.21.15.32:0/1040038832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 -- 172.21.15.32:0/1040038832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 msgr2=0x560617adcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617adcf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 -- 172.21.15.32:0/1040038832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ad9800 msgr2=0x560617b4dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ad9800 0x560617b4dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.320+0000 7f6702ffd640 1 -- 172.21.15.32:0/1040038832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560617aced70 con 0x560617ace620 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f6702ffd640 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617adc9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x560617ad7530 tx=0x7f66f802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9543533299386ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8035030 con 0x560617ace620 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f66f8059040 con 0x560617ace620 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66f401f040 con 0x560617ace620 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f66f8004490 con 0x560617ace620 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 10 client.26717 fetch_fsmap learned FSMap version 50 2024-04-23T18:29:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x560617b4f3c0 con 0x560617ace620 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f80038d0 con 0x560617ace620 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8342+0+0 (secure 0 0 0) 0x7f66f8046040 con 0x560617ace620 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 10 client.26717 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 10 client.26717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 20 client.26717 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617b519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 4 client.26717 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.321+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f66f401f040 con 0x560617ae9840 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.322+0000 7f67037fe640 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617b519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.322+0000 7f66e2ffd640 10 client.26717 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.322+0000 7f67037fe640 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617b519b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3c2d941eb82fcc8 server_cookie=a3e892cfe45d53cc in_seq=0 out_seq=0 2024-04-23T18:29:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.334+0000 7f66e2ffd640 1 -- 172.21.15.32:0/1040038832 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f66f401f040 con 0x560617ae9840 2024-04-23T18:29:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.334+0000 7f66e2ffd640 10 client.26717 handle_command_reply: tid=0 2024-04-23T18:29:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.334+0000 7f670c981740 1 client.26717 shutdown 2024-04-23T18:29:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.334+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 msgr2=0x560617adc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.334+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617adc9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x560617ad7530 tx=0x7f66f802a480 comp rx=0 tx=0).stop 2024-04-23T18:29:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 shutdown_connections 2024-04-23T18:29:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x560617ae9840 0x560617b519b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560617ad9800 0x560617b4dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f66e2ffd640 0 client.26717 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560617acef40 0x560617adcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 --2- 172.21.15.32:0/1040038832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560617ace620 0x560617adc9d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.335+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 >> 172.21.15.32:0/1040038832 conn(0x560617acab30 msgr2=0x560617ad8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.336+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 shutdown_connections 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.336+0000 7f670c981740 1 -- 172.21.15.32:0/1040038832 wait complete. 2024-04-23T18:29:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.336+0000 7f670c981740 20 client.26717 trim_cache size 0 max 16384 2024-04-23T18:29:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:29:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:29:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:29:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:29:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.336+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 msgr2=0x7f66ec0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 0x7f66ec0a6250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66fc099760 tx=0x7f66fc0057c0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 msgr2=0x7f670419cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 0x7f670419cff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009fd0 tx=0x7f66fc00ef10 comp rx=0 tx=0).stop 2024-04-23T18:29:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 shutdown_connections 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f670410c590 0x7f670419cff0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6704103ae0 0x7f6704198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f66ec0a3d90 0x7f66ec0a6250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 --2- 172.21.15.32:0/2668629693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041031c0 0x7f6704198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 >> 172.21.15.32:0/2668629693 conn(0x7f6704077fa0 msgr2=0x7f67040fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 shutdown_connections 2024-04-23T18:29:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:43.337+0000 7f6709c74640 1 -- 172.21.15.32:0/2668629693 wait complete. 2024-04-23T18:29:43.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:43 smithi064 ceph-mon[33062]: pgmap v458: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:43.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:43 smithi032 ceph-mon[29221]: pgmap v458: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:43.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:43 smithi040 ceph-mon[32626]: pgmap v458: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:43.860 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 855.0 s, 2256.0 tps, lat 43.972 ms stddev 52.112 2024-04-23T18:29:45.417 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:45 smithi040 ceph-mon[32626]: pgmap v459: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:45.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:45 smithi064 ceph-mon[33062]: pgmap v459: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:45.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:45 smithi032 ceph-mon[29221]: pgmap v459: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 MiB/s wr, 604 op/s 2024-04-23T18:29:47.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:47 smithi064 ceph-mon[33062]: pgmap v460: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 23 MiB/s wr, 886 op/s 2024-04-23T18:29:47.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:47 smithi032 ceph-mon[29221]: pgmap v460: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 23 MiB/s wr, 886 op/s 2024-04-23T18:29:47.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:47 smithi040 ceph-mon[32626]: pgmap v460: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 23 MiB/s wr, 886 op/s 2024-04-23T18:29:48.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:48 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:48.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:48 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:48.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:48 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:48.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 860.0 s, 2004.0 tps, lat 49.957 ms stddev 59.152 2024-04-23T18:29:49.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:49 smithi032 ceph-mon[29221]: pgmap v461: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-23T18:29:49.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:49 smithi064 ceph-mon[33062]: pgmap v461: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-23T18:29:49.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:49 smithi040 ceph-mon[32626]: pgmap v461: 97 pgs: 97 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-23T18:29:51.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:51 smithi064 ceph-mon[33062]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-04-23T18:29:51.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:51 smithi032 ceph-mon[29221]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-04-23T18:29:51.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:51 smithi040 ceph-mon[32626]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-04-23T18:29:53.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:53 smithi064 ceph-mon[33062]: pgmap v463: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:53.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:53 smithi032 ceph-mon[29221]: pgmap v463: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:53.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:53 smithi040 ceph-mon[32626]: pgmap v463: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:53.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 865.0 s, 1990.0 tps, lat 50.350 ms stddev 56.209 2024-04-23T18:29:55.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:55 smithi064 ceph-mon[33062]: pgmap v464: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:55.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:55 smithi032 ceph-mon[29221]: pgmap v464: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:55.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:55 smithi040 ceph-mon[32626]: pgmap v464: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-04-23T18:29:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 -- 172.21.15.32:0/3468893803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a4107270 msgr2=0x7f08a410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/3468893803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a4107270 0x7f08a410ee40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f089c009ca0 tx=0x7f089c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 -- 172.21.15.32:0/3468893803 shutdown_connections 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/3468893803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a4107270 0x7f08a410ee40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/3468893803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a4102b50 0x7f08a4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/3468893803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a41021a0 0x7f08a4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 -- 172.21.15.32:0/3468893803 >> 172.21.15.32:0/3468893803 conn(0x7f08a40fda40 msgr2=0x7f08a40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 -- 172.21.15.32:0/3468893803 shutdown_connections 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.459+0000 7f08aaa7c640 1 -- 172.21.15.32:0/3468893803 wait complete. 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 Processor -- start 2024-04-23T18:29:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 -- start start 2024-04-23T18:29:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a41021a0 0x7f08a4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a419cfc0 0x7f08a419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4112210 con 0x7f08a41021a0 2024-04-23T18:29:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4112090 con 0x7f08a419cfc0 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f08aaa7c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4112390 con 0x7f08a4107270 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f089bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f089bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:34586/0 (socket says 172.21.15.32:34586) 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.460+0000 7f089bfff640 1 -- 172.21.15.32:0/879079204 learned_addr learned my addr 172.21.15.32:0/879079204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f08a9019640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a419cfc0 0x7f08a419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 -- 172.21.15.32:0/879079204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a419cfc0 msgr2=0x7f08a419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a419cfc0 0x7f08a419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 -- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a41021a0 msgr2=0x7f08a4198750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a41021a0 0x7f08a4198750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 -- 172.21.15.32:0/879079204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a419dab0 con 0x7f08a4107270 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f089bfff640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f089400b2c0 tx=0x7f089400b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f0899ffb640 1 -- 172.21.15.32:0/879079204 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894005020 con 0x7f08a4107270 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f0899ffb640 1 -- 172.21.15.32:0/879079204 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0894005670 con 0x7f08a4107270 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a419dda0 con 0x7f08a4107270 2024-04-23T18:29:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.461+0000 7f0899ffb640 1 -- 172.21.15.32:0/879079204 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089401b780 con 0x7f08a4107270 2024-04-23T18:29:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.462+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a40744f0 con 0x7f08a4107270 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.464+0000 7f08ad793740 1 Processor -- start 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.464+0000 7f08ad793740 1 -- start start 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.464+0000 7f08ad793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef84a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08ad793740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08ad793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef91bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08ad793740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56270eee9970 con 0x56270ef84620 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08ad793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56270ee864a0 con 0x56270ef8f800 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08ad793740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56270eee5930 con 0x56270ef84f40 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a9019640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:34602/0 (socket says 172.21.15.32:34602) 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 -- 172.21.15.32:0/2620352669 learned_addr learned my addr 172.21.15.32:0/2620352669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f0899ffb640 1 -- 172.21.15.32:0/879079204 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f089401b9e0 con 0x7f08a4107270 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 -- 172.21.15.32:0/2620352669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 msgr2=0x56270ef91bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef91bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 -- 172.21.15.32:0/2620352669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 msgr2=0x56270ef84a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef84a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 -- 172.21.15.32:0/2620352669 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270ebfc210 con 0x56270ef84f40 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f08a8818640 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009bd0 tx=0x7f08a000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2d1e8eafb01e12 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f088affd640 1 -- 172.21.15.32:0/2620352669 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0005020 con 0x56270ef84f40 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f088affd640 1 -- 172.21.15.32:0/2620352669 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a00047a0 con 0x56270ef84f40 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.465+0000 7f088affd640 1 -- 172.21.15.32:0/2620352669 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0010670 con 0x56270ef84f40 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f0899ffb640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 0x7f088c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08a9019640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 0x7f088c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f0899ffb640 1 -- 172.21.15.32:0/879079204 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f08940d86c0 con 0x7f08a4107270 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 -- 172.21.15.32:0/2620352669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 msgr2=0x56270ef8f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009bd0 tx=0x7f08a000e720 comp rx=0 tx=0).stop 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08a9019640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 0x7f088c0a6250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f089c02f540 tx=0x7f089c02f3b0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 -- 172.21.15.32:0/2620352669 shutdown_connections 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef91bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270ef8f2c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 --2- 172.21.15.32:0/2620352669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef84a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.466+0000 7f08ad793740 1 -- 172.21.15.32:0/2620352669 >> 172.21.15.32:0/2620352669 conn(0x56270ef80b30 msgr2=0x56270ef80f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.467+0000 7f08ad793740 1 -- 172.21.15.32:0/2620352669 shutdown_connections 2024-04-23T18:29:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.467+0000 7f08ad793740 1 -- 172.21.15.32:0/2620352669 wait complete. 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.468+0000 7f08ad793740 1 -- start start 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.468+0000 7f08ad793740 1 Processor -- start 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84f40 0x56270ef8c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef8c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08a8818640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08a8818640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:34614/0 (socket says 172.21.15.32:34614) 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08a8818640 1 -- 172.21.15.32:0/670295892 learned_addr learned my addr 172.21.15.32:0/670295892 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56270ef92430 con 0x56270ef84f40 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08a9019640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef8c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56270ee840b0 con 0x56270ef8f800 2024-04-23T18:29:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56270ef925b0 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f089bfff640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84f40 0x56270ef8c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.469+0000 7f08ad793740 10 client.0 _ll_register_callbacks cb 0x56270ee71d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 -- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 msgr2=0x56270ef8c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef8c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 -- 172.21.15.32:0/670295892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84f40 msgr2=0x56270ef8c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84f40 0x56270ef8c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270ecc8620 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08a8818640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f08a00097c0 tx=0x7f08a0009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd5b1dbd9cb5aa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f0883fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a001d040 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f08a0005320 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08a00047a0 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f08a0030050 con 0x56270ef84620 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08ad793740 10 client.26758 fetch_fsmap learned FSMap version 51 2024-04-23T18:29:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x56270ef9e0e0 con 0x56270ef84620 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.470+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a00047a0 con 0x56270ef84620 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f08a0010430 con 0x56270ef84620 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 10 client.26758 fetch_fsmap finished waiting for FSMap version 51 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 10 client.26758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 20 client.26758 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270efa1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 4 client.26758 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56270eef7390 con 0x56270ef9f490 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f089bfff640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270efa1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f0883fff640 10 client.26758 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.471+0000 7f089bfff640 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270efa1880 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ef71264e1d76ecc server_cookie=33d4b13dd22eae63 in_seq=0 out_seq=0 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f0883fff640 1 -- 172.21.15.32:0/670295892 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56270eef7390 con 0x56270ef9f490 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f0883fff640 10 client.26758 handle_command_reply: tid=0 2024-04-23T18:29:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 client.26758 shutdown 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 msgr2=0x56270ef8f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f08a00097c0 tx=0x7f08a0009910 comp rx=0 tx=0).stop 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 shutdown_connections 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270efa1880 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f0883fff640 0 client.26758 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef8f800 0x56270ef8c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84f40 0x56270ef8c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 --2- 172.21.15.32:0/670295892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84620 0x56270ef8f110 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 >> 172.21.15.32:0/670295892 conn(0x56270ef80b30 msgr2=0x56270ef868b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 shutdown_connections 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 1 -- 172.21.15.32:0/670295892 wait complete. 2024-04-23T18:29:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.472+0000 7f08ad793740 20 client.26758 trim_cache size 0 max 16384 2024-04-23T18:29:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.491+0000 7f08ad793740 1 Processor -- start 2024-04-23T18:29:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.491+0000 7f08ad793740 1 -- start start 2024-04-23T18:29:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.491+0000 7f08ad793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84620 0x56270ef9b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08ad793740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270efa6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08ad793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08ad793740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41ae1f0 con 0x56270ef8f800 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08ad793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41abac0 con 0x56270ef84620 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08ad793740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41a9910 con 0x56270ef84f40 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a8818640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84620 0x56270ef9b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57110/0 (socket says 172.21.15.32:57110) 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 -- 172.21.15.32:0/1423009825 learned_addr learned my addr 172.21.15.32:0/1423009825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f089bfff640 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270efa6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 -- 172.21.15.32:0/1423009825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 msgr2=0x56270efa6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270efa6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 -- 172.21.15.32:0/1423009825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84620 msgr2=0x56270ef9b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84620 0x56270ef9b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 -- 172.21.15.32:0/1423009825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270ef7b090 con 0x56270ef8f800 2024-04-23T18:29:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.492+0000 7f08a9019640 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f089c009ca0 tx=0x7f089c044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f1d74f9719ac8f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.493+0000 7f088a7fc640 1 -- 172.21.15.32:0/1423009825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c057070 con 0x56270ef8f800 2024-04-23T18:29:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.493+0000 7f088a7fc640 1 -- 172.21.15.32:0/1423009825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f089c036040 con 0x56270ef8f800 2024-04-23T18:29:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.493+0000 7f088a7fc640 1 -- 172.21.15.32:0/1423009825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c030ac0 con 0x56270ef8f800 2024-04-23T18:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- 172.21.15.32:0/1423009825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 msgr2=0x56270efa7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f089c009ca0 tx=0x7f089c044000 comp rx=0 tx=0).stop 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- 172.21.15.32:0/1423009825 shutdown_connections 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef8f800 0x56270efa7390 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef84f40 0x56270efa6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 --2- 172.21.15.32:0/1423009825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84620 0x56270ef9b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- 172.21.15.32:0/1423009825 >> 172.21.15.32:0/1423009825 conn(0x56270ef80b30 msgr2=0x56270ef7f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- 172.21.15.32:0/1423009825 shutdown_connections 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- 172.21.15.32:0/1423009825 wait complete. 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 -- start start 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.494+0000 7f08ad793740 1 Processor -- start 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57122/0 (socket says 172.21.15.32:57122) 2024-04-23T18:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84f40 0x56270ef97d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 -- 172.21.15.32:0/35617303 learned_addr learned my addr 172.21.15.32:0/35617303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef8f800 0x56270ef94de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41a9f80 con 0x56270ef84620 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41ab3f0 con 0x56270ef84f40 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41aa650 con 0x56270ef8f800 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f089bfff640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84f40 0x56270ef97d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08ad793740 10 client.0 _ll_register_callbacks cb 0x7f08a0005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a9019640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef8f800 0x56270ef94de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 -- 172.21.15.32:0/35617303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef8f800 msgr2=0x56270ef94de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef8f800 0x56270ef94de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 -- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84f40 msgr2=0x56270ef97d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84f40 0x56270ef97d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.495+0000 7f08a8818640 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270ef955e0 con 0x56270ef84620 2024-04-23T18:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08a8818640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f08a0004600 tx=0x7f08a0004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f982897fec3f32f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0038070 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8440+0+0 (secure 0 0 0) 0x7f08a0005000 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f089c036040 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f08a0044050 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 10 client.17253 fetch_fsmap learned FSMap version 51 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x56270f005350 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f089c036040 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f08a00345c0 con 0x56270ef84620 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 10 client.17253 fetch_fsmap finished waiting for FSMap version 51 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 20 client.17253 populate_metadata read hostname 'smithi032' 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270f007a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 4 client.17253 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.496+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56270eef1200 con 0x56270ef9f490 2024-04-23T18:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.497+0000 7f089bfff640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270f007a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.497+0000 7f089bfff640 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270f007a80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8129f0c9cc818b4 server_cookie=919377650abb9310 in_seq=0 out_seq=0 2024-04-23T18:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.497+0000 7f0883fff640 10 client.17253 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.510+0000 7f0883fff640 1 -- 172.21.15.32:0/35617303 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56270eef1200 con 0x56270ef9f490 2024-04-23T18:29:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.510+0000 7f0883fff640 10 client.17253 handle_command_reply: tid=0 2024-04-23T18:29:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.510+0000 7f08ad793740 1 client.17253 shutdown 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.510+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 msgr2=0x56270ef97850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f08a0004600 tx=0x7f08a0004860 comp rx=0 tx=0).stop 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 shutdown_connections 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56270ef9f490 0x56270f007a80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56270ef8f800 0x56270ef94de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56270ef84f40 0x56270ef97d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 --2- 172.21.15.32:0/35617303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270ef84620 0x56270ef97850 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f0883fff640 0 client.17253 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.511+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 >> 172.21.15.32:0/35617303 conn(0x56270ef80b30 msgr2=0x56270ef7fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 shutdown_connections 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08ad793740 1 -- 172.21.15.32:0/35617303 wait complete. 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08ad793740 20 client.17253 trim_cache size 0 max 16384 2024-04-23T18:29:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 msgr2=0x7f088c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 0x7f088c0a6250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f089c02f540 tx=0x7f089c02f3b0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 msgr2=0x7f08a4198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f089400b2c0 tx=0x7f089400b780 comp rx=0 tx=0).stop 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 shutdown_connections 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a419cfc0 0x7f08a419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08a4107270 0x7f08a4198c90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f088c0a3d90 0x7f088c0a6250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 --2- 172.21.15.32:0/879079204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a41021a0 0x7f08a4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.512+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 >> 172.21.15.32:0/879079204 conn(0x7f08a40fda40 msgr2=0x7f08a40ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.513+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 shutdown_connections 2024-04-23T18:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:29:56.513+0000 7f08aaa7c640 1 -- 172.21.15.32:0/879079204 wait complete. 2024-04-23T18:29:57.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:57 smithi064 ceph-mon[33062]: pgmap v465: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 806 op/s 2024-04-23T18:29:57.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:57 smithi032 ceph-mon[29221]: pgmap v465: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 806 op/s 2024-04-23T18:29:57.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:57 smithi040 ceph-mon[32626]: pgmap v465: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 806 op/s 2024-04-23T18:29:58.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 870.0 s, 1650.0 tps, lat 61.013 ms stddev 69.095 2024-04-23T18:29:59.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:29:59 smithi064 ceph-mon[33062]: pgmap v466: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-23T18:29:59.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:29:59 smithi032 ceph-mon[29221]: pgmap v466: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-23T18:29:59.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:29:59 smithi040 ceph-mon[32626]: pgmap v466: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-23T18:30:00.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:00 smithi032 ceph-mon[29221]: overall HEALTH_OK 2024-04-23T18:30:00.618 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:00 smithi040 ceph-mon[32626]: overall HEALTH_OK 2024-04-23T18:30:00.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:00 smithi064 ceph-mon[33062]: overall HEALTH_OK 2024-04-23T18:30:01.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:01 smithi032 ceph-mon[29221]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 910 op/s 2024-04-23T18:30:01.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:01 smithi040 ceph-mon[32626]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 910 op/s 2024-04-23T18:30:01.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:01 smithi064 ceph-mon[33062]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 910 op/s 2024-04-23T18:30:02.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:02 smithi032 ceph-mon[29221]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:02.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:02 smithi040 ceph-mon[32626]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:02.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:02 smithi064 ceph-mon[33062]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 875.0 s, 929.4 tps, lat 102.926 ms stddev 149.111 2024-04-23T18:30:05.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:04 smithi032 ceph-mon[29221]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:05.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:04 smithi040 ceph-mon[32626]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:05.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:04 smithi064 ceph-mon[33062]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-23T18:30:07.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:06 smithi032 ceph-mon[29221]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-23T18:30:07.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:06 smithi040 ceph-mon[32626]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-23T18:30:07.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:06 smithi064 ceph-mon[33062]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-23T18:30:08.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:07 smithi032 ceph-mon[29221]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:30:08.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:07 smithi040 ceph-mon[32626]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:30:08.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:07 smithi064 ceph-mon[33062]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:30:08.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 880.0 s, 482.0 tps, lat 207.586 ms stddev 322.013 2024-04-23T18:30:09.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:08 smithi032 ceph-mon[29221]: pgmap v472: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 709 op/s 2024-04-23T18:30:09.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:08 smithi040 ceph-mon[32626]: pgmap v472: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 709 op/s 2024-04-23T18:30:09.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:08 smithi064 ceph-mon[33062]: pgmap v472: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 709 op/s 2024-04-23T18:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.634+0000 7f5896079640 1 -- 172.21.15.32:0/3115966343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 msgr2=0x7f589010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.634+0000 7f5896079640 1 --2- 172.21.15.32:0/3115966343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f589010dd40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5888009be0 tx=0x7f588802a6c0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 -- 172.21.15.32:0/3115966343 shutdown_connections 2024-04-23T18:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 --2- 172.21.15.32:0/3115966343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f589010dd40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 --2- 172.21.15.32:0/3115966343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5890101a50 0x7f5890105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 --2- 172.21.15.32:0/3115966343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58901010a0 0x7f5890101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 -- 172.21.15.32:0/3115966343 >> 172.21.15.32:0/3115966343 conn(0x7f5890077f70 msgr2=0x7f5890078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 -- 172.21.15.32:0/3115966343 shutdown_connections 2024-04-23T18:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.635+0000 7f5896079640 1 -- 172.21.15.32:0/3115966343 wait complete. 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.636+0000 7f5896079640 1 Processor -- start 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.636+0000 7f5896079640 1 -- start start 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.636+0000 7f5896079640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f5896079640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f58901a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f5896079640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 0x7f589019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f5896079640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f589010fdb0 con 0x7f58901010a0 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f5896079640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f589010fc30 con 0x7f5890106170 2024-04-23T18:30:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f5896079640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f589010ff30 con 0x7f589019a710 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f58901a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43246/0 (socket says 172.21.15.32:43246) 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 -- 172.21.15.32:0/991513101 learned_addr learned my addr 172.21.15.32:0/991513101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588ffff640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 0x7f589019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 -- 172.21.15.32:0/991513101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 msgr2=0x7f589019ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 0x7f589019ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 -- 172.21.15.32:0/991513101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 msgr2=0x7f58901a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f58901a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588f7fe640 1 -- 172.21.15.32:0/991513101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589019b310 con 0x7f58901010a0 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588ffff640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 0x7f589019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.637+0000 7f588effd640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f58901a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:09.632 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f588f7fe640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9f0 tx=0x7f588400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f588cff9640 1 -- 172.21.15.32:0/991513101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588400cd80 con 0x7f58901010a0 2024-04-23T18:30:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58901a73f0 con 0x7f58901010a0 2024-04-23T18:30:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901a7850 con 0x7f58901010a0 2024-04-23T18:30:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f588cff9640 1 -- 172.21.15.32:0/991513101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58840108e0 con 0x7f58901010a0 2024-04-23T18:30:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.638+0000 7f588cff9640 1 -- 172.21.15.32:0/991513101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884018980 con 0x7f58901010a0 2024-04-23T18:30:09.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.641+0000 7f5898d8e740 1 Processor -- start 2024-04-23T18:30:09.634 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 -- start start 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e5922c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e594bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d82e4ec970 con 0x55d82e587f40 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d82e4894a0 con 0x55d82e592800 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f5898d8e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d82e4e8930 con 0x55d82e587620 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f588ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f588effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e594bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f588ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:56338/0 (socket says 172.21.15.32:56338) 2024-04-23T18:30:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.642+0000 7f588ffff640 1 -- 172.21.15.32:0/665403421 learned_addr learned my addr 172.21.15.32:0/665403421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 -- 172.21.15.32:0/665403421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 msgr2=0x55d82e594bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e594bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 -- 172.21.15.32:0/665403421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 msgr2=0x55d82e5922c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e5922c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 -- 172.21.15.32:0/665403421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82e1ff210 con 0x55d82e587620 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588ffff640 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5888009be0 tx=0x7f58880314c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501594de316bfaea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f588cff9640 1 -- 172.21.15.32:0/991513101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5884010430 con 0x7f58901010a0 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f5875ffb640 1 -- 172.21.15.32:0/665403421 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888042070 con 0x55d82e587620 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f5875ffb640 1 -- 172.21.15.32:0/665403421 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5888037040 con 0x55d82e587620 2024-04-23T18:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.643+0000 7f5875ffb640 1 -- 172.21.15.32:0/665403421 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888036030 con 0x55d82e587620 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f588cff9640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 0x7f58780a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f588cff9640 1 -- 172.21.15.32:0/991513101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8528+0+0 (secure 0 0 0) 0x7f58840d08f0 con 0x7f58901010a0 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 -- 172.21.15.32:0/665403421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 msgr2=0x55d82e587a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5888009be0 tx=0x7f58880314c0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 -- 172.21.15.32:0/665403421 shutdown_connections 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e594bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e5922c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 --2- 172.21.15.32:0/665403421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e587a00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.644+0000 7f5898d8e740 1 -- 172.21.15.32:0/665403421 >> 172.21.15.32:0/665403421 conn(0x55d82e583b30 msgr2=0x55d82e583f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.645+0000 7f588f7fe640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 0x7f58780a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.645+0000 7f5898d8e740 1 -- 172.21.15.32:0/665403421 shutdown_connections 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.645+0000 7f5898d8e740 1 -- 172.21.15.32:0/665403421 wait complete. 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.645+0000 7f588f7fe640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 0x7f58780a6250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d82e588bc0 tx=0x7f5884009310 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.645+0000 7f5898d8e740 1 -- start start 2024-04-23T18:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 Processor -- start 2024-04-23T18:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e592130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e58f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d82e5953d0 con 0x55d82e587f40 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d82e4870b0 con 0x55d82e592800 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d82e595550 con 0x55d82e587620 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f5898d8e740 10 client.0 _ll_register_callbacks cb 0x55d82e474d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e592130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e58f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43256/0 (socket says 172.21.15.32:43256) 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 -- 172.21.15.32:0/1151815164 learned_addr learned my addr 172.21.15.32:0/1151815164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 -- 172.21.15.32:0/1151815164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 msgr2=0x55d82e592130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e592130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 -- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 msgr2=0x55d82e58f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.646+0000 7f588effd640 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e58f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f588effd640 1 -- 172.21.15.32:0/1151815164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82e2cb620 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f588effd640 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5880005fd0 tx=0x7f588000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fcac38472e2e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f586effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588001f060 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5888037040 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8528+0+0 (secure 0 0 0) 0x7f5880010430 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5880010630 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f5898d8e740 10 client.17262 fetch_fsmap learned FSMap version 51 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.647+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d82e5a1430 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5888037040 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f588001a820 con 0x55d82e587f40 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 10 client.17262 fetch_fsmap finished waiting for FSMap version 51 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 10 client.17262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 20 client.17262 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e5a28f0 0x55d82e5a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 4 client.17262 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d82e4fa390 con 0x55d82e5a28f0 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f588ffff640 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e5a28f0 0x55d82e5a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f586effd640 10 client.17262 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.648+0000 7f588ffff640 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e5a28f0 0x55d82e5a4d00 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b301bd2c2250f786 server_cookie=b3ed6c88012dbd10 in_seq=0 out_seq=0 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.650+0000 7f586effd640 1 -- 172.21.15.32:0/1151815164 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d82e4fa390 con 0x55d82e5a28f0 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.650+0000 7f586effd640 10 client.17262 handle_command_reply: tid=0 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.650+0000 7f5898d8e740 1 client.17262 shutdown 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 msgr2=0x55d82e58f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5880005fd0 tx=0x7f588000f790 comp rx=0 tx=0).stop 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 shutdown_connections 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e5a28f0 0x55d82e5a4d00 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e592800 0x55d82e58f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e58f420 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1151815164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e592130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 >> 172.21.15.32:0/1151815164 conn(0x55d82e583b30 msgr2=0x55d82e589860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f586effd640 0 client.17262 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.651+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 shutdown_connections 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.652+0000 7f5898d8e740 1 -- 172.21.15.32:0/1151815164 wait complete. 2024-04-23T18:30:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.652+0000 7f5898d8e740 20 client.17262 trim_cache size 0 max 16384 2024-04-23T18:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.673+0000 7f5898d8e740 1 Processor -- start 2024-04-23T18:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.673+0000 7f5898d8e740 1 -- start start 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e583b30 0x55d82e59e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e591c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f587c011050 con 0x55d82e587f40 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f587c00e920 con 0x55d82e583b30 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f5898d8e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f587c00c770 con 0x55d82e587620 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e583b30 0x55d82e59e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:56360/0 (socket says 172.21.15.32:56360) 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588effd640 1 -- 172.21.15.32:0/1175300228 learned_addr learned my addr 172.21.15.32:0/1175300228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e591c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 -- 172.21.15.32:0/1175300228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 msgr2=0x55d82e58a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 -- 172.21.15.32:0/1175300228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e583b30 msgr2=0x55d82e59e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e583b30 0x55d82e59e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 -- 172.21.15.32:0/1175300228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82e590810 con 0x55d82e587f40 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588effd640 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.674+0000 7f588ffff640 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e591c60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5888009560 tx=0x7f5888005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823189fcfbf069a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f58757fa640 1 -- 172.21.15.32:0/1175300228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888042040 con 0x55d82e587f40 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f58757fa640 1 -- 172.21.15.32:0/1175300228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5888031b90 con 0x55d82e587f40 2024-04-23T18:30:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f58757fa640 1 -- 172.21.15.32:0/1175300228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888036040 con 0x55d82e587f40 2024-04-23T18:30:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 -- 172.21.15.32:0/1175300228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 msgr2=0x55d82e591c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e591c60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5888009560 tx=0x7f5888005fd0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 -- 172.21.15.32:0/1175300228 shutdown_connections 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e587f40 0x55d82e591c60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587620 0x55d82e58a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 --2- 172.21.15.32:0/1175300228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e583b30 0x55d82e59e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 -- 172.21.15.32:0/1175300228 >> 172.21.15.32:0/1175300228 conn(0x55d82e58fac0 msgr2=0x55d82e58a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 -- 172.21.15.32:0/1175300228 shutdown_connections 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.675+0000 7f5898d8e740 1 -- 172.21.15.32:0/1175300228 wait complete. 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 -- start start 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 Processor -- start 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 0x55d82e59aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 0x55d82e58bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587f40 0x55d82e58c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f587c00cde0 con 0x55d82e583b30 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f587c00e250 con 0x55d82e587620 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f587c00d4b0 con 0x55d82e587f40 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f5898d8e740 10 client.0 _ll_register_callbacks cb 0x7f58800128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f588ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 0x55d82e59aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f588ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 0x55d82e59aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43274/0 (socket says 172.21.15.32:43274) 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f588ffff640 1 -- 172.21.15.32:0/4123606874 learned_addr learned my addr 172.21.15.32:0/4123606874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.676+0000 7f588effd640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 0x55d82e58bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588ffff640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587f40 0x55d82e58c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 -- 172.21.15.32:0/4123606874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587f40 msgr2=0x55d82e58c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587f40 0x55d82e58c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 -- 172.21.15.32:0/4123606874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 msgr2=0x55d82e59aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 0x55d82e59aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 -- 172.21.15.32:0/4123606874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82e58cab0 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f588effd640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 0x55d82e58bd90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5880000c00 tx=0x7f58800059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52be7e2be5a25831 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f586effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880032060 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8528+0+0 (secure 0 0 0) 0x7f588001a610 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.677+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5888031b90 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f588001a850 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 10 client.26750 fetch_fsmap learned FSMap version 51 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d82e608de0 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5888031b90 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f5880010b70 con 0x55d82e587620 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 10 client.26750 fetch_fsmap finished waiting for FSMap version 51 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 10 client.26750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 20 client.26750 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e592800 0x55d82e60b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 4 client.26750 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d82e4f4200 con 0x55d82e592800 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.678+0000 7f588ffff640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e592800 0x55d82e60b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.679+0000 7f586effd640 10 client.26750 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.679+0000 7f588ffff640 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e592800 0x55d82e60b500 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec547e218cfb5ebc server_cookie=2937547e3a3b93ff in_seq=0 out_seq=0 2024-04-23T18:30:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f586effd640 1 -- 172.21.15.32:0/4123606874 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d82e4f4200 con 0x55d82e592800 2024-04-23T18:30:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f586effd640 10 client.26750 handle_command_reply: tid=0 2024-04-23T18:30:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 client.26750 shutdown 2024-04-23T18:30:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 msgr2=0x55d82e58bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 0x55d82e58bd90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5880000c00 tx=0x7f58800059e0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 shutdown_connections 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55d82e592800 0x55d82e60b500 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d82e587f40 0x55d82e58c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d82e587620 0x55d82e58bd90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f586effd640 0 client.26750 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 --2- 172.21.15.32:0/4123606874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d82e583b30 0x55d82e59aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 >> 172.21.15.32:0/4123606874 conn(0x55d82e58fac0 msgr2=0x55d82e589860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 shutdown_connections 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 1 -- 172.21.15.32:0/4123606874 wait complete. 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.692+0000 7f5898d8e740 20 client.26750 trim_cache size 0 max 16384 2024-04-23T18:30:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:30:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:30:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:30:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:30:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 msgr2=0x7f58780a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 0x7f58780a6250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d82e588bc0 tx=0x7f5884009310 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 msgr2=0x7f58901a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9f0 tx=0x7f588400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 shutdown_connections 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f589019a710 0x7f589019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5890106170 0x7f58901a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f58780a3d90 0x7f58780a6250 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 --2- 172.21.15.32:0/991513101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58901010a0 0x7f58901a0650 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 >> 172.21.15.32:0/991513101 conn(0x7f5890077f70 msgr2=0x7f58900febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 shutdown_connections 2024-04-23T18:30:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:09.693+0000 7f5896079640 1 -- 172.21.15.32:0/991513101 wait complete. 2024-04-23T18:30:11.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:10 smithi032 ceph-mon[29221]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 26 MiB/s wr, 1.91k op/s 2024-04-23T18:30:11.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:10 smithi040 ceph-mon[32626]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 26 MiB/s wr, 1.91k op/s 2024-04-23T18:30:11.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:10 smithi064 ceph-mon[33062]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 26 MiB/s wr, 1.91k op/s 2024-04-23T18:30:12.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:11 smithi032 ceph-mon[29221]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:30:12.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:11 smithi064 ceph-mon[33062]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:30:12.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:11 smithi040 ceph-mon[32626]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:30:13.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:12 smithi032 ceph-mon[29221]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-23T18:30:13.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:12 smithi040 ceph-mon[32626]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-23T18:30:13.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:12 smithi064 ceph-mon[33062]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-23T18:30:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 885.0 s, 1783.3 tps, lat 58.199 ms stddev 69.579 2024-04-23T18:30:15.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:14 smithi040 ceph-mon[32626]: pgmap v476: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 2.18k op/s 2024-04-23T18:30:15.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:14 smithi064 ceph-mon[33062]: pgmap v476: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 2.18k op/s 2024-04-23T18:30:15.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:14 smithi032 ceph-mon[29221]: pgmap v476: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 2.18k op/s 2024-04-23T18:30:17.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:16 smithi040 ceph-mon[32626]: pgmap v477: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 2.31k op/s 2024-04-23T18:30:17.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:16 smithi064 ceph-mon[33062]: pgmap v477: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 2.31k op/s 2024-04-23T18:30:17.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:16 smithi032 ceph-mon[29221]: pgmap v477: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 2.31k op/s 2024-04-23T18:30:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 890.0 s, 2178.2 tps, lat 45.842 ms stddev 51.299 2024-04-23T18:30:19.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:18 smithi032 ceph-mon[29221]: pgmap v478: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 29 MiB/s wr, 2.06k op/s 2024-04-23T18:30:19.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:18 smithi040 ceph-mon[32626]: pgmap v478: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 29 MiB/s wr, 2.06k op/s 2024-04-23T18:30:19.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:18 smithi064 ceph-mon[33062]: pgmap v478: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 29 MiB/s wr, 2.06k op/s 2024-04-23T18:30:21.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:20 smithi040 ceph-mon[32626]: pgmap v479: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 10 MiB/s wr, 715 op/s 2024-04-23T18:30:21.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:20 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:21.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:20 smithi064 ceph-mon[33062]: pgmap v479: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 10 MiB/s wr, 715 op/s 2024-04-23T18:30:21.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:20 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:20 smithi032 ceph-mon[29221]: pgmap v479: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 10 MiB/s wr, 715 op/s 2024-04-23T18:30:21.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:20 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 -- 172.21.15.32:0/443000910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101ce0 msgr2=0x7f1024105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 --2- 172.21.15.32:0/443000910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101ce0 0x7f1024105af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1018009860 tx=0x7f101802a340 comp rx=0 tx=0).stop 2024-04-23T18:30:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 -- 172.21.15.32:0/443000910 shutdown_connections 2024-04-23T18:30:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 --2- 172.21.15.32:0/443000910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f102410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 --2- 172.21.15.32:0/443000910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101ce0 0x7f1024105af0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 --2- 172.21.15.32:0/443000910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1024101330 0x7f1024101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 -- 172.21.15.32:0/443000910 >> 172.21.15.32:0/443000910 conn(0x7f1024077d30 msgr2=0x7f1024078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 -- 172.21.15.32:0/443000910 shutdown_connections 2024-04-23T18:30:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.826+0000 7f1028c49640 1 -- 172.21.15.32:0/443000910 wait complete. 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.827+0000 7f1028c49640 1 Processor -- start 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.827+0000 7f1028c49640 1 -- start start 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f1024197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f102419bf30 0x7f102419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10241110f0 con 0x7f102419bf30 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1024110f70 con 0x7f1024101330 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1028c49640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1024111270 con 0x7f10241061d0 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f102259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f1024197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f102259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54384/0 (socket says 172.21.15.32:54384) 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f102259c640 1 -- 172.21.15.32:0/1192317940 learned_addr learned my addr 172.21.15.32:0/1192317940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1022d9d640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f102419bf30 0x7f102419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 -- 172.21.15.32:0/1192317940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 msgr2=0x7f1024197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 -- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f102419bf30 msgr2=0x7f102419c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f102419bf30 0x7f102419c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 -- 172.21.15.32:0/1192317940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102419ca20 con 0x7f10241061d0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f102259c640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.828+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f1024197bb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1018009780 tx=0x7f1018032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.829+0000 7f10137fe640 1 -- 172.21.15.32:0/1192317940 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018037040 con 0x7f10241061d0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.829+0000 7f10137fe640 1 -- 172.21.15.32:0/1192317940 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1018030c10 con 0x7f10241061d0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.829+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102419ccb0 con 0x7f10241061d0 2024-04-23T18:30:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.829+0000 7f10137fe640 1 -- 172.21.15.32:0/1192317940 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180035f0 con 0x7f10241061d0 2024-04-23T18:30:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.829+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102406a390 con 0x7f10241061d0 2024-04-23T18:30:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.831+0000 7f102ba33740 1 Processor -- start 2024-04-23T18:30:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.831+0000 7f10137fe640 1 -- 172.21.15.32:0/1192317940 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1018043020 con 0x7f10241061d0 2024-04-23T18:30:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.832+0000 7f10137fe640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 0x7f100c0a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.833+0000 7f1022d9d640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 0x7f100c0a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.826 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.833+0000 7f10137fe640 1 -- 172.21.15.32:0/1192317940 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f10180efae0 con 0x7f10241061d0 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 -- start start 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aa620 0x556cc57aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556cc570f970 con 0x556cc57b5800 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556cc56ac4a0 con 0x556cc57aaf40 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f102ba33740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556cc570b930 con 0x556cc57aa620 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54396/0 (socket says 172.21.15.32:54396) 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.834+0000 7f1021d9b640 1 -- 172.21.15.32:0/3807329786 learned_addr learned my addr 172.21.15.32:0/3807329786 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f102259c640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aa620 0x556cc57aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1022d9d640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 -- 172.21.15.32:0/3807329786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aa620 msgr2=0x556cc57aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aa620 0x556cc57aaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 -- 172.21.15.32:0/3807329786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 msgr2=0x556cc57b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 -- 172.21.15.32:0/3807329786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc5422210 con 0x556cc57aaf40 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1022d9d640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1021d9b640 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f10180c4290 tx=0x7f1018030fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ae8859a23a8fba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1022d9d640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 0x7f100c0a62d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f101c005fd0 tx=0x7f101c011810 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1010ff9640 1 -- 172.21.15.32:0/3807329786 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180328c0 con 0x556cc57aaf40 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1010ff9640 1 -- 172.21.15.32:0/3807329786 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1018030dc0 con 0x556cc57aaf40 2024-04-23T18:30:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.835+0000 7f1010ff9640 1 -- 172.21.15.32:0/3807329786 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180dc180 con 0x556cc57aaf40 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 -- 172.21.15.32:0/3807329786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 msgr2=0x556cc57b52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f10180c4290 tx=0x7f1018030fc0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 -- 172.21.15.32:0/3807329786 shutdown_connections 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b52c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 --2- 172.21.15.32:0/3807329786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aa620 0x556cc57aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 -- 172.21.15.32:0/3807329786 >> 172.21.15.32:0/3807329786 conn(0x556cc57a6b30 msgr2=0x556cc57a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 -- 172.21.15.32:0/3807329786 shutdown_connections 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.836+0000 7f102ba33740 1 -- 172.21.15.32:0/3807329786 wait complete. 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 -- start start 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 Processor -- start 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556cc56ac4a0 con 0x556cc57aa620 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556cc56aa0b0 con 0x556cc57aaf40 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50956/0 (socket says 172.21.15.32:50956) 2024-04-23T18:30:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102259c640 1 -- 172.21.15.32:0/1258453429 learned_addr learned my addr 172.21.15.32:0/1258453429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f1022d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556cc57c9200 con 0x556cc57b5800 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.837+0000 7f102ba33740 10 client.0 _ll_register_callbacks cb 0x556cc5697d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 -- 172.21.15.32:0/1258453429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 msgr2=0x556cc57b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 -- 172.21.15.32:0/1258453429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 msgr2=0x556cc57b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 -- 172.21.15.32:0/1258453429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc54ee620 con 0x556cc57aaf40 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1022d9d640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f102259c640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1021d9b640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b2b30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556cc57abbc0 tx=0x7f101802a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d28169e912a56ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:30:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180ed680 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f10180c4330 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1018030dc0 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f10180bbb50 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f102ba33740 10 client.26762 fetch_fsmap learned FSMap version 52 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x556cc57d5070 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.838+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1018030dc0 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f10180baa10 con 0x556cc57aaf40 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 10 client.26762 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 10 client.26762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 20 client.26762 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc57d61f0 0x556cc57d85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 4 client.26762 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cc571d390 con 0x556cc57d61f0 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102259c640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc57d61f0 0x556cc57d85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f102259c640 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc57d61f0 0x556cc57d85e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3367d3047cff04d4 server_cookie=c29997c35911ff77 in_seq=0 out_seq=0 2024-04-23T18:30:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.839+0000 7f1009ffb640 10 client.26762 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.841+0000 7f1009ffb640 1 -- 172.21.15.32:0/1258453429 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cc571d390 con 0x556cc57d61f0 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.841+0000 7f1009ffb640 10 client.26762 handle_command_reply: tid=0 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.842+0000 7f102ba33740 1 client.26762 shutdown 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 msgr2=0x556cc57b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b2b30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556cc57abbc0 tx=0x7f101802a340 comp rx=0 tx=0).stop 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 shutdown_connections 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc57d61f0 0x556cc57d85e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aaf40 0x556cc57b2b30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 --2- 172.21.15.32:0/1258453429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aa620 0x556cc57b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f1009ffb640 0 client.26762 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 >> 172.21.15.32:0/1258453429 conn(0x556cc57a6b30 msgr2=0x556cc57ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 shutdown_connections 2024-04-23T18:30:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 1 -- 172.21.15.32:0/1258453429 wait complete. 2024-04-23T18:30:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.843+0000 7f102ba33740 20 client.26762 trim_cache size 0 max 16384 2024-04-23T18:30:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 Processor -- start 2024-04-23T18:30:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 -- start start 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 0x556cc57b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57dde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10241a59f0 con 0x556cc57aaf40 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1024073230 con 0x556cc57aa620 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102ba33740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10240710a0 con 0x556cc57b5800 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 0x556cc57b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f102259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 0x556cc57b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50968/0 (socket says 172.21.15.32:50968) 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f1021d9b640 1 -- 172.21.15.32:0/109158278 learned_addr learned my addr 172.21.15.32:0/109158278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.868+0000 7f1022d9d640 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57dde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 -- 172.21.15.32:0/109158278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 msgr2=0x556cc57dde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57dde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 -- 172.21.15.32:0/109158278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 msgr2=0x556cc57b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 0x556cc57b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 -- 172.21.15.32:0/109158278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc57d8950 con 0x556cc57aa620 2024-04-23T18:30:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f102259c640 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57d1f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1014009fd0 tx=0x7f1014010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a32022f49935d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f100bfff640 1 -- 172.21.15.32:0/109158278 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014004020 con 0x556cc57aa620 2024-04-23T18:30:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.869+0000 7f100bfff640 1 -- 172.21.15.32:0/109158278 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101401e8e0 con 0x556cc57aa620 2024-04-23T18:30:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f100bfff640 1 -- 172.21.15.32:0/109158278 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140047a0 con 0x556cc57aa620 2024-04-23T18:30:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 -- 172.21.15.32:0/109158278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 msgr2=0x556cc57d1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57d1f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1014009fd0 tx=0x7f1014010800 comp rx=0 tx=0).stop 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 -- 172.21.15.32:0/109158278 shutdown_connections 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57b5800 0x556cc57dde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57aaf40 0x556cc57b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 --2- 172.21.15.32:0/109158278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57d1f40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 -- 172.21.15.32:0/109158278 >> 172.21.15.32:0/109158278 conn(0x556cc57a6b30 msgr2=0x556cc57b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 -- 172.21.15.32:0/109158278 shutdown_connections 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.870+0000 7f102ba33740 1 -- 172.21.15.32:0/109158278 wait complete. 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 -- start start 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 Processor -- start 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aaf40 0x556cc57cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10240716f0 con 0x556cc57b5800 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1024072b60 con 0x556cc57aa620 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1024071dc0 con 0x556cc57aaf40 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f1022d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aaf40 0x556cc57cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54420/0 (socket says 172.21.15.32:54420) 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 -- 172.21.15.32:0/2186132156 learned_addr learned my addr 172.21.15.32:0/2186132156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102ba33740 10 client.0 _ll_register_callbacks cb 0x7f10180b7ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 -- 172.21.15.32:0/2186132156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aaf40 msgr2=0x556cc57cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aaf40 0x556cc57cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 -- 172.21.15.32:0/2186132156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 msgr2=0x556cc57cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.871+0000 7f102259c640 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f102259c640 1 -- 172.21.15.32:0/2186132156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc583a870 con 0x556cc57aa620 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f102259c640 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1014005490 tx=0x7f10140054c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e34bc86a7b7a2992 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f1009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101401e630 con 0x556cc57aa620 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f101401e8e0 con 0x556cc57aa620 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f1014013b10 con 0x556cc57aa620 2024-04-23T18:30:22.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f10140056e0 con 0x556cc57aa620 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f102ba33740 10 client.26768 fetch_fsmap learned FSMap version 52 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.872+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x556cc583bb20 con 0x556cc57aa620 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.874+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101401e8e0 con 0x556cc57aa620 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f101402d020 con 0x556cc57aa620 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 10 client.26768 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 20 client.26768 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc583c1b0 0x556cc583e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:22.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 4 client.26768 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556cc5717200 con 0x556cc583c1b0 2024-04-23T18:30:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.875+0000 7f1021d9b640 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc583c1b0 0x556cc583e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.876+0000 7f1021d9b640 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc583c1b0 0x556cc583e5d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26e74a3219d3b3a8 server_cookie=b27439f70f44eaa3 in_seq=0 out_seq=0 2024-04-23T18:30:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.876+0000 7f1009ffb640 10 client.26768 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:22.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.913+0000 7f1009ffb640 1 -- 172.21.15.32:0/2186132156 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556cc5717200 con 0x556cc583c1b0 2024-04-23T18:30:22.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.913+0000 7f1009ffb640 10 client.26768 handle_command_reply: tid=0 2024-04-23T18:30:22.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.913+0000 7f102ba33740 1 client.26768 shutdown 2024-04-23T18:30:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 msgr2=0x556cc57cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1014005490 tx=0x7f10140054c0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 shutdown_connections 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x556cc583c1b0 0x556cc583e5d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556cc57b5800 0x556cc57cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc57aaf40 0x556cc57cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f1009ffb640 0 client.26768 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 --2- 172.21.15.32:0/2186132156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556cc57aa620 0x556cc57cb420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 >> 172.21.15.32:0/2186132156 conn(0x556cc57a6b30 msgr2=0x556cc57a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 shutdown_connections 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 1 -- 172.21.15.32:0/2186132156 wait complete. 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.914+0000 7f102ba33740 20 client.26768 trim_cache size 0 max 16384 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:30:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 msgr2=0x7f100c0a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 0x7f100c0a62d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f101c005fd0 tx=0x7f101c011810 comp rx=0 tx=0).stop 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 msgr2=0x7f1024197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f1024197bb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1018009780 tx=0x7f1018032d50 comp rx=0 tx=0).stop 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 shutdown_connections 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f100c0a3e10 0x7f100c0a62d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f102419bf30 0x7f102419c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10241061d0 0x7f1024197bb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 --2- 172.21.15.32:0/1192317940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101330 0x7f1024197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 >> 172.21.15.32:0/1192317940 conn(0x7f1024077d30 msgr2=0x7f10240ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 shutdown_connections 2024-04-23T18:30:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:22.915+0000 7f1028c49640 1 -- 172.21.15.32:0/1192317940 wait complete. 2024-04-23T18:30:23.062 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:30:23.062 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.85 2024-04-23T18:30:23.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:23.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:22 smithi032 ceph-mon[29221]: pgmap v480: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 9.5 MiB/s wr, 656 op/s 2024-04-23T18:30:23.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:22 smithi040 ceph-mon[32626]: pgmap v480: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 9.5 MiB/s wr, 656 op/s 2024-04-23T18:30:23.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:22 smithi064 ceph-mon[33062]: pgmap v480: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 9.5 MiB/s wr, 656 op/s 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.229+0000 7f4292c7c640 1 -- 172.21.15.32:0/3448622795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 msgr2=0x7f428c107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.229+0000 7f4292c7c640 1 --2- 172.21.15.32:0/3448622795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c107290 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4284009ca0 tx=0x7f428402a950 comp rx=0 tx=0).stop 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 -- 172.21.15.32:0/3448622795 shutdown_connections 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 --2- 172.21.15.32:0/3448622795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c105ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 --2- 172.21.15.32:0/3448622795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 0x7f428c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 --2- 172.21.15.32:0/3448622795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c107290 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 -- 172.21.15.32:0/3448622795 >> 172.21.15.32:0/3448622795 conn(0x7f428c0773a0 msgr2=0x7f428c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 -- 172.21.15.32:0/3448622795 shutdown_connections 2024-04-23T18:30:23.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.230+0000 7f4292c7c640 1 -- 172.21.15.32:0/3448622795 wait complete. 2024-04-23T18:30:23.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.231+0000 7f4292c7c640 1 Processor -- start 2024-04-23T18:30:23.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.231+0000 7f4292c7c640 1 -- start start 2024-04-23T18:30:23.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 0x7f428c19ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c1a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f428c118840 con 0x7f428c106eb0 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f428c1186c0 con 0x7f428c100450 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4292c7c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f428c1189c0 con 0x7f428c100df0 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f428bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c1a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f428bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c19f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49526/0 (socket says 172.21.15.32:49526) 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f428bfff640 1 -- 172.21.15.32:0/215324852 learned_addr learned my addr 172.21.15.32:0/215324852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4290a18640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 0x7f428c19ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 -- 172.21.15.32:0/215324852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 msgr2=0x7f428c19f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c19f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 -- 172.21.15.32:0/215324852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 msgr2=0x7f428c19ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 0x7f428c19ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.232+0000 7f4291219640 1 -- 172.21.15.32:0/215324852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c1a3d10 con 0x7f428c106eb0 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.233+0000 7f4291219640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c1a35e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f428000d680 tx=0x7f428000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.233+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280017070 con 0x7f428c106eb0 2024-04-23T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.234+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c1a4000 con 0x7f428c106eb0 2024-04-23T18:30:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.234+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4280009820 con 0x7f428c106eb0 2024-04-23T18:30:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.235+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280004ce0 con 0x7f428c106eb0 2024-04-23T18:30:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.235+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c1ab890 con 0x7f428c106eb0 2024-04-23T18:30:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.236+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c101670 con 0x7f428c106eb0 2024-04-23T18:30:23.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.237+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4280005220 con 0x7f428c106eb0 2024-04-23T18:30:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.238+0000 7f4289ffb640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 0x7f42740a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.239+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f42800cfbd0 con 0x7f428c106eb0 2024-04-23T18:30:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.239+0000 7f4290a18640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 0x7f42740a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.240+0000 7f4290a18640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 0x7f42740a6200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f80 tx=0x7f4284009820 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.247+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f428009bbe0 con 0x7f428c106eb0 2024-04-23T18:30:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.600+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f428c1a0070 con 0x7f428c106eb0 2024-04-23T18:30:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.605+0000 7f4289ffb640 1 -- 172.21.15.32:0/215324852 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11852 (secure 0 0 0) 0x7f428009fee0 con 0x7f428c106eb0 2024-04-23T18:30:23.599 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:30:23.599 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:30:18.812216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:30:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-04-23T18:30:23.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.610+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 msgr2=0x7f42740a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.610+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 0x7f42740a6200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f80 tx=0x7f4284009820 comp rx=0 tx=0).stop 2024-04-23T18:30:23.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.610+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 msgr2=0x7f428c1a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.610+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c1a35e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f428000d680 tx=0x7f428000db40 comp rx=0 tx=0).stop 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 shutdown_connections 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f42740a3d40 0x7f42740a6200 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428c106eb0 0x7f428c1a35e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f428c100df0 0x7f428c19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 --2- 172.21.15.32:0/215324852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f428c100450 0x7f428c19ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.611+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 >> 172.21.15.32:0/215324852 conn(0x7f428c0773a0 msgr2=0x7f428c0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.612+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 shutdown_connections 2024-04-23T18:30:23.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.614+0000 7f4292c7c640 1 -- 172.21.15.32:0/215324852 wait complete. 2024-04-23T18:30:23.627 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 84.85 --await-for 42.0' 2024-04-23T18:30:23.627 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 84.85 --await-for 42.0 2024-04-23T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.839+0000 7f56661bf640 1 -- 172.21.15.32:0/1972034127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5660101a40 msgr2=0x7f566010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.839+0000 7f56661bf640 1 --2- 172.21.15.32:0/1972034127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5660101a40 0x7f566010dc70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5658009ca0 tx=0x7f565802a8b0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 -- 172.21.15.32:0/1972034127 shutdown_connections 2024-04-23T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 --2- 172.21.15.32:0/1972034127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5660101a40 0x7f566010dc70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 --2- 172.21.15.32:0/1972034127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56601010a0 0x7f5660101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 --2- 172.21.15.32:0/1972034127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 0x7f56601072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 -- 172.21.15.32:0/1972034127 >> 172.21.15.32:0/1972034127 conn(0x7f5660077c40 msgr2=0x7f5660078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.840+0000 7f56661bf640 1 -- 172.21.15.32:0/1972034127 shutdown_connections 2024-04-23T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.841+0000 7f56661bf640 1 -- 172.21.15.32:0/1972034127 wait complete. 2024-04-23T18:30:23.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.841+0000 7f56661bf640 1 Processor -- start 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.841+0000 7f56661bf640 1 -- start start 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 0x7f56601986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 0x7f5660198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 0x7f566019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5660112200 con 0x7f5660106ed0 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5660112080 con 0x7f56601010a0 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f56661bf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5660112380 con 0x7f5660101a40 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 0x7f56601986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 0x7f56601986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54468/0 (socket says 172.21.15.32:54468) 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f565f7fe640 1 -- 172.21.15.32:0/3398312903 learned_addr learned my addr 172.21.15.32:0/3398312903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f565effd640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 0x7f5660198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.842+0000 7f565ffff640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 0x7f566019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 -- 172.21.15.32:0/3398312903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 msgr2=0x7f56601986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 0x7f56601986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 -- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 msgr2=0x7f566019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 0x7f566019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 -- 172.21.15.32:0/3398312903 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566019d6b0 con 0x7f5660101a40 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565effd640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 0x7f5660198c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f565400e9f0 tx=0x7f565400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565cff9640 1 -- 172.21.15.32:0/3398312903 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565400ccc0 con 0x7f5660101a40 2024-04-23T18:30:23.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565cff9640 1 -- 172.21.15.32:0/3398312903 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5654005020 con 0x7f5660101a40 2024-04-23T18:30:23.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.843+0000 7f565cff9640 1 -- 172.21.15.32:0/3398312903 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654004a00 con 0x7f5660101a40 2024-04-23T18:30:23.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.844+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566019d9a0 con 0x7f5660101a40 2024-04-23T18:30:23.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.844+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566010e330 con 0x7f5660101a40 2024-04-23T18:30:23.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.846+0000 7f565cff9640 1 -- 172.21.15.32:0/3398312903 <== mon.1 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5654004ba0 con 0x7f5660101a40 2024-04-23T18:30:23.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.847+0000 7f565cff9640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 0x7f56480aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.847+0000 7f565cff9640 1 -- 172.21.15.32:0/3398312903 <== mon.1 v2:172.21.15.64:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56540d06b0 con 0x7f5660101a40 2024-04-23T18:30:23.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.847+0000 7f565f7fe640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 0x7f56480aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.847+0000 7f5668fb1740 1 Processor -- start 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.847+0000 7f5668fb1740 1 -- start start 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7513b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 0x7f56480aa7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5650009c10 tx=0x7f5650005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c7516480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5583c746e370 con 0x5583c75099c0 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5583c740b2e0 con 0x5583c75090a0 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f5668fb1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5583c746a330 con 0x5583c7514090 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7513b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7513b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54482/0 (socket says 172.21.15.32:54482) 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565effd640 1 -- 172.21.15.32:0/577504890 learned_addr learned my addr 172.21.15.32:0/577504890 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565ffff640 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c7516480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 -- 172.21.15.32:0/577504890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 msgr2=0x5583c7516480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c7516480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 -- 172.21.15.32:0/577504890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 msgr2=0x5583c7513b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7513b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 -- 172.21.15.32:0/577504890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583c724cd80 con 0x5583c75090a0 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.848+0000 7f565f7fe640 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5650009ad0 tx=0x7f5650014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=677b2b2cf0b43466 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5645ffb640 1 -- 172.21.15.32:0/577504890 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500265f0 con 0x5583c75090a0 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5645ffb640 1 -- 172.21.15.32:0/577504890 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5650026c40 con 0x5583c75090a0 2024-04-23T18:30:23.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5645ffb640 1 -- 172.21.15.32:0/577504890 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565000ed20 con 0x5583c75090a0 2024-04-23T18:30:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 -- 172.21.15.32:0/577504890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 msgr2=0x5583c7509480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5650009ad0 tx=0x7f5650014000 comp rx=0 tx=0).stop 2024-04-23T18:30:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 -- 172.21.15.32:0/577504890 shutdown_connections 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c7516480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7513b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 --2- 172.21.15.32:0/577504890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c7509480 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 -- 172.21.15.32:0/577504890 >> 172.21.15.32:0/577504890 conn(0x5583c75053c0 msgr2=0x5583c75057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 -- 172.21.15.32:0/577504890 shutdown_connections 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.849+0000 7f5668fb1740 1 -- 172.21.15.32:0/577504890 wait complete. 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.850+0000 7f5668fb1740 1 -- start start 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.850+0000 7f5668fb1740 1 Processor -- start 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7511010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c7514090 0x5583c7511a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5583c7516ed0 con 0x5583c75099c0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5583c7408fb0 con 0x5583c7514090 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5583c7517050 con 0x5583c75090a0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f5668fb1740 10 client.0 _ll_register_callbacks cb 0x5583c74089a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f565effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f565effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51042/0 (socket says 172.21.15.32:51042) 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f565effd640 1 -- 172.21.15.32:0/1979129531 learned_addr learned my addr 172.21.15.32:0/1979129531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.851+0000 7f565f7fe640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7511010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565ffff640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c7514090 0x5583c7511a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 -- 172.21.15.32:0/1979129531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 msgr2=0x5583c7511010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7511010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 -- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c7514090 msgr2=0x5583c7511a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c7514090 0x5583c7511a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 -- 172.21.15.32:0/1979129531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583c72ad510 con 0x5583c75099c0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f565effd640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f565400c6a0 tx=0x7f565409c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e584ac433c83d1ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f563effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540ba060 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56540a4df0 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5650026c40 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.852+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f565401c030 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 10 client.17301 fetch_fsmap learned FSMap version 52 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5583c7140670 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5650026c40 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f56540bb250 con 0x5583c75099c0 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 10 client.17301 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 10 client.17301 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 20 client.17301 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c752cb40 0x5583c752ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 4 client.17301 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583c747bd90 con 0x5583c752cb40 2024-04-23T18:30:23.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.853+0000 7f565f7fe640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c752cb40 0x5583c752ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.854+0000 7f565f7fe640 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c752cb40 0x5583c752ef30 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c5ac2d9c426a08 server_cookie=74be08627d3a93f7 in_seq=0 out_seq=0 2024-04-23T18:30:23.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.854+0000 7f563effd640 10 client.17301 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.854+0000 7f563effd640 1 -- 172.21.15.32:0/1979129531 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583c747bd90 con 0x5583c752cb40 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.854+0000 7f563effd640 10 client.17301 handle_command_reply: tid=0 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.854+0000 7f5668fb1740 1 client.17301 shutdown 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 msgr2=0x5583c7511550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f565400c6a0 tx=0x7f565409c030 comp rx=0 tx=0).stop 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 shutdown_connections 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c752cb40 0x5583c752ef30 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c7514090 0x5583c7511a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c7511550 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f563effd640 0 client.17301 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 --2- 172.21.15.32:0/1979129531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7511010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 >> 172.21.15.32:0/1979129531 conn(0x5583c75053c0 msgr2=0x5583c750b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 shutdown_connections 2024-04-23T18:30:23.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 1 -- 172.21.15.32:0/1979129531 wait complete. 2024-04-23T18:30:23.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.855+0000 7f5668fb1740 20 client.17301 trim_cache size 0 max 16384 2024-04-23T18:30:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 895.0 s, 1961.8 tps, lat 51.151 ms stddev 90.613 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 Processor -- start 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 -- start start 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c75116e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c75347e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56601ae200 con 0x5583c75099c0 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56601abaa0 con 0x5583c75090a0 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f5668fb1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56601a9910 con 0x5583c7514090 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:54510/0 (socket says 172.21.15.32:54510) 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f565f7fe640 1 -- 172.21.15.32:0/3490308304 learned_addr learned my addr 172.21.15.32:0/3490308304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f565effd640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c75116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.878+0000 7f565ffff640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c75347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 -- 172.21.15.32:0/3490308304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 msgr2=0x5583c75347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c75347e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 -- 172.21.15.32:0/3490308304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 msgr2=0x5583c75116e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c75116e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 -- 172.21.15.32:0/3490308304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583c750ffc0 con 0x5583c75090a0 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565effd640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c75116e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.879+0000 7f565f7fe640 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f565000e890 tx=0x7f565000e8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19c66b21fc2db9eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.880+0000 7f56457fa640 1 -- 172.21.15.32:0/3490308304 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565003d070 con 0x5583c75090a0 2024-04-23T18:30:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.880+0000 7f56457fa640 1 -- 172.21.15.32:0/3490308304 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5650004030 con 0x5583c75090a0 2024-04-23T18:30:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.880+0000 7f56457fa640 1 -- 172.21.15.32:0/3490308304 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650005040 con 0x5583c75090a0 2024-04-23T18:30:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 -- 172.21.15.32:0/3490308304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 msgr2=0x5583c750b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f565000e890 tx=0x7f565000e8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 -- 172.21.15.32:0/3490308304 shutdown_connections 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c7514090 0x5583c75347e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c75099c0 0x5583c75116e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 --2- 172.21.15.32:0/3490308304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75090a0 0x5583c750b800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 -- 172.21.15.32:0/3490308304 >> 172.21.15.32:0/3490308304 conn(0x5583c75053c0 msgr2=0x5583c7510fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 -- 172.21.15.32:0/3490308304 shutdown_connections 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.881+0000 7f5668fb1740 1 -- 172.21.15.32:0/3490308304 wait complete. 2024-04-23T18:30:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.882+0000 7f5668fb1740 1 -- start start 2024-04-23T18:30:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.882+0000 7f5668fb1740 1 Processor -- start 2024-04-23T18:30:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 0x5583c7521b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c7514090 0x5583c7522040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56601a9f60 con 0x5583c7514090 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56601ab3d0 con 0x5583c75099c0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56601aa630 con 0x5583c75090a0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f5668fb1740 10 client.0 _ll_register_callbacks cb 0x7f565409c490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:49584/0 (socket says 172.21.15.32:49584) 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f565effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 0x5583c7521b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f565f7fe640 1 -- 172.21.15.32:0/723925358 learned_addr learned my addr 172.21.15.32:0/723925358 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.883+0000 7f565ffff640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c7514090 0x5583c7522040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 -- 172.21.15.32:0/723925358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 msgr2=0x5583c7524ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 -- 172.21.15.32:0/723925358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c7514090 msgr2=0x5583c7522040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c7514090 0x5583c7522040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 -- 172.21.15.32:0/723925358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583c75051f0 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565f7fe640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f565effd640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 0x5583c7521b00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5654098fe0 tx=0x7f56540053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13d1a56b8240298e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f563effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540a5280 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5650004030 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56540a4df0 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5654082050 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f5668fb1740 10 client.26792 fetch_fsmap learned FSMap version 52 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5583c7593dd0 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.884+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5650004030 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f565409b4d0 con 0x5583c75099c0 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 10 client.26792 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 10 client.26792 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 20 client.26792 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c75945c0 0x5583c75969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 4 client.26792 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.85, "await_for": 42.0}] 2024-04-23T18:30:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.85, "await_for": 42.0}) -- 0x5583c7475c00 con 0x5583c75945c0 2024-04-23T18:30:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.885+0000 7f565f7fe640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c75945c0 0x5583c75969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.886+0000 7f565f7fe640 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c75945c0 0x5583c75969b0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b2dcb7caa9745e server_cookie=5b22053bc0849b6c in_seq=0 out_seq=0 2024-04-23T18:30:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:23.887+0000 7f563effd640 10 client.26792 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:24.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:23 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/215324852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:24.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:23 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/215324852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f563effd640 1 -- 172.21.15.32:0/723925358 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5583c7475c00 con 0x5583c75945c0 2024-04-23T18:30:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f563effd640 10 client.26792 handle_command_reply: tid=0 2024-04-23T18:30:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 client.26792 shutdown 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 msgr2=0x5583c7521b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 0x5583c7521b00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5654098fe0 tx=0x7f56540053b0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 shutdown_connections 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5583c75945c0 0x5583c75969b0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583c7514090 0x5583c7522040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583c75099c0 0x5583c7521b00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.127+0000 7f563effd640 0 client.26792 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 1 --2- 172.21.15.32:0/723925358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583c75090a0 0x5583c7524ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 >> 172.21.15.32:0/723925358 conn(0x5583c75053c0 msgr2=0x5583c7511860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 shutdown_connections 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 1 -- 172.21.15.32:0/723925358 wait complete. 2024-04-23T18:30:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f5668fb1740 20 client.26792 trim_cache size 0 max 16384 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 52, 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 50, 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "fd2e910d": { 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 50, 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:24.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 84.8, 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:24.123 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:30:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 msgr2=0x7f56480aa7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 0x7f56480aa7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5650009c10 tx=0x7f5650005f90 comp rx=0 tx=0).stop 2024-04-23T18:30:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 msgr2=0x7f5660198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 0x7f5660198c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f565400e9f0 tx=0x7f565400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 shutdown_connections 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56480a8300 0x7f56480aa7c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5660106ed0 0x7f566019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5660101a40 0x7f5660198c00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 --2- 172.21.15.32:0/3398312903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56601010a0 0x7f56601986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.128+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 >> 172.21.15.32:0/3398312903 conn(0x7f5660077c40 msgr2=0x7f56600ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.129+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 shutdown_connections 2024-04-23T18:30:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:24.129+0000 7f56661bf640 1 -- 172.21.15.32:0/3398312903 wait complete. 2024-04-23T18:30:24.132 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fd2e910d', quiesce took 1.1 seconds. Will release after: 24.85 2024-04-23T18:30:24.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:23 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/215324852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:25.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:24 smithi040 ceph-mon[32626]: pgmap v481: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 10 MiB/s wr, 776 op/s 2024-04-23T18:30:25.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:24 smithi064 ceph-mon[33062]: pgmap v481: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 10 MiB/s wr, 776 op/s 2024-04-23T18:30:25.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:24 smithi032 ceph-mon[29221]: pgmap v481: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 10 MiB/s wr, 776 op/s 2024-04-23T18:30:26.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:25 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:26.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:25 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:26.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:25 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:27.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:26 smithi040 ceph-mon[32626]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 21 MiB/s wr, 1.67k op/s 2024-04-23T18:30:27.215 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:26 smithi064 ceph-mon[33062]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 21 MiB/s wr, 1.67k op/s 2024-04-23T18:30:27.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:26 smithi032 ceph-mon[29221]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 21 MiB/s wr, 1.67k op/s 2024-04-23T18:30:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 900.0 s, 38.2 tps, lat 26.615 ms stddev 30.790 2024-04-23T18:30:29.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:28 smithi040 ceph-mon[32626]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 MiB/s wr, 1.41k op/s 2024-04-23T18:30:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:28 smithi064 ceph-mon[33062]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 MiB/s wr, 1.41k op/s 2024-04-23T18:30:29.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:28 smithi032 ceph-mon[29221]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 MiB/s wr, 1.41k op/s 2024-04-23T18:30:31.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:30 smithi040 ceph-mon[32626]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-23T18:30:31.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:30 smithi064 ceph-mon[33062]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-23T18:30:31.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:30 smithi032 ceph-mon[29221]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-23T18:30:33.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:32 smithi040 ceph-mon[32626]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-04-23T18:30:33.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:32 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:33.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:32 smithi064 ceph-mon[33062]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-04-23T18:30:33.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:32 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:32 smithi032 ceph-mon[29221]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-04-23T18:30:33.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:32 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:33.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 905.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:30:34.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:33 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:34.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:33 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:34.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:33 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:30:34.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:33 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:34.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:33 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:34.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:33 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:30:34.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:33 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:34.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:33 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:34.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:33 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:30:35.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:35 smithi032 ceph-mon[29221]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-04-23T18:30:35.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:35 smithi040 ceph-mon[32626]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-04-23T18:30:35.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:35 smithi064 ceph-mon[33062]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-04-23T18:30:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 -- 172.21.15.32:0/1990435451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 msgr2=0x7f1dd8101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/1990435451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd8101120 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009900 tx=0x7f1dd402a410 comp rx=0 tx=0).stop 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 -- 172.21.15.32:0/1990435451 shutdown_connections 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/1990435451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd810e8f0 0x7f1dd8110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/1990435451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd8101660 0x7f1dd810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/1990435451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd8101120 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 -- 172.21.15.32:0/1990435451 >> 172.21.15.32:0/1990435451 conn(0x7f1dd8077f70 msgr2=0x7f1dd8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 -- 172.21.15.32:0/1990435451 shutdown_connections 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.022+0000 7f1ddf34f640 1 -- 172.21.15.32:0/1990435451 wait complete. 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 Processor -- start 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 -- start start 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd8101660 0x7f1dd81a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd810e8f0 0x7f1dd819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81143f0 con 0x7f1dd8100d40 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8114270 con 0x7f1dd810e8f0 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddf34f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8114570 con 0x7f1dd8101660 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53286/0 (socket says 172.21.15.32:53286) 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/3661927880 learned_addr learned my addr 172.21.15.32:0/3661927880 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd8ec640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd810e8f0 0x7f1dd819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd8101660 0x7f1dd81a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/3661927880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd8101660 msgr2=0x7f1dd81a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd8101660 0x7f1dd81a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/3661927880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd810e8f0 msgr2=0x7f1dd819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd810e8f0 0x7f1dd819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.023+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/3661927880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd819afe0 con 0x7f1dd8100d40 2024-04-23T18:30:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4005940 tx=0x7f1dd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1dce7fc640 1 -- 172.21.15.32:0/3661927880 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd403f030 con 0x7f1dd8100d40 2024-04-23T18:30:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1dce7fc640 1 -- 172.21.15.32:0/3661927880 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd4003cb0 con 0x7f1dd8100d40 2024-04-23T18:30:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd819b270 con 0x7f1dd8100d40 2024-04-23T18:30:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1dce7fc640 1 -- 172.21.15.32:0/3661927880 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4035070 con 0x7f1dd8100d40 2024-04-23T18:30:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.024+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd81a74f0 con 0x7f1dd8100d40 2024-04-23T18:30:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1dce7fc640 1 -- 172.21.15.32:0/3661927880 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1dd4042030 con 0x7f1dd8100d40 2024-04-23T18:30:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 Processor -- start 2024-04-23T18:30:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 -- start start 2024-04-23T18:30:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56119e2ff970 con 0x56119e3a5800 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56119e29c4a0 con 0x56119e39a620 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.027+0000 7f1de206d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56119e2fb930 con 0x56119e39af40 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1dce7fc640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 0x7f1dc40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53294/0 (socket says 172.21.15.32:53294) 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/4198815607 learned_addr learned my addr 172.21.15.32:0/4198815607 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddd8ec640 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddd8ec640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 0x7f1dc40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1dce7fc640 1 -- 172.21.15.32:0/3661927880 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f1dd40f1060 con 0x7f1dd8100d40 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/4198815607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 msgr2=0x56119e3a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/4198815607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 msgr2=0x56119e39aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/4198815607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56119e012210 con 0x56119e3a5800 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800bc40 tx=0x7f1dc800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe2ac4735b0d2b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.028+0000 7f1ddd8ec640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 0x7f1dc40a61d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009fd0 tx=0x7f1dd0009600 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1dc37fe640 1 -- 172.21.15.32:0/4198815607 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8015070 con 0x56119e3a5800 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1dc37fe640 1 -- 172.21.15.32:0/4198815607 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc801a910 con 0x56119e3a5800 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1dc37fe640 1 -- 172.21.15.32:0/4198815607 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8011830 con 0x56119e3a5800 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 -- 172.21.15.32:0/4198815607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 msgr2=0x56119e3a7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800bc40 tx=0x7f1dc800f730 comp rx=0 tx=0).stop 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 -- 172.21.15.32:0/4198815607 shutdown_connections 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e3a5800 0x56119e3a7bf0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 --2- 172.21.15.32:0/4198815607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 -- 172.21.15.32:0/4198815607 >> 172.21.15.32:0/4198815607 conn(0x56119e396b30 msgr2=0x56119e396f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.029+0000 7f1de206d740 1 -- 172.21.15.32:0/4198815607 shutdown_connections 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 -- 172.21.15.32:0/4198815607 wait complete. 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 -- start start 2024-04-23T18:30:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 Processor -- start 2024-04-23T18:30:36.028 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e3a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56119e3a8430 con 0x56119e39af40 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56119e29a0b0 con 0x56119e39a620 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56119e3a85b0 con 0x56119e3a5800 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1de206d740 10 client.0 _ll_register_callbacks cb 0x56119e287d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.030+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53310/0 (socket says 172.21.15.32:53310) 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/1116533519 learned_addr learned my addr 172.21.15.32:0/1116533519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddd8ec640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e3a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/1116533519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 msgr2=0x56119e3a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 msgr2=0x56119e3a50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e3a50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/1116533519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56119e0de620 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e3a50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8005d80 tx=0x7f1dc800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1896f92fdd27f2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1dc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8015070 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f1dc801aac0 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc801a910 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1dc801ace0 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1de206d740 10 client.17319 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.031+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x56119e3b42d0 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc801a910 con 0x56119e39af40 2024-04-23T18:30:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f1dc802a030 con 0x56119e39af40 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 10 client.17319 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 10 client.17319 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 20 client.17319 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e3b56d0 0x56119e3b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 4 client.17319 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56119e30d390 con 0x56119e3b56d0 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.032+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e3b56d0 0x56119e3b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.033+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e3b56d0 0x56119e3b7ac0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2ab38edc3b7c8d server_cookie=5f8a632deec29de5 in_seq=0 out_seq=0 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.033+0000 7f1dc0ff9640 10 client.17319 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.033+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/1116533519 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56119e30d390 con 0x56119e3b56d0 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.033+0000 7f1dc0ff9640 10 client.17319 handle_command_reply: tid=0 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.033+0000 7f1de206d740 1 client.17319 shutdown 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 msgr2=0x56119e3a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8005d80 tx=0x7f1dc800b8e0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 shutdown_connections 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e3b56d0 0x56119e3b7ac0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a23c0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 --2- 172.21.15.32:0/1116533519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e3a50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1dc0ff9640 0 client.17319 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 >> 172.21.15.32:0/1116533519 conn(0x56119e396b30 msgr2=0x56119e39c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 shutdown_connections 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 1 -- 172.21.15.32:0/1116533519 wait complete. 2024-04-23T18:30:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.034+0000 7f1de206d740 20 client.17319 trim_cache size 0 max 16384 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 Processor -- start 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 -- start start 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3bd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81b2660 con 0x56119e39af40 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81afed0 con 0x56119e39a620 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1de206d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81add40 con 0x56119e3a5800 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53316/0 (socket says 172.21.15.32:53316) 2024-04-23T18:30:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/2359511773 learned_addr learned my addr 172.21.15.32:0/2359511773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/2359511773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 msgr2=0x56119e3bd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3bd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/2359511773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 msgr2=0x56119e39daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddc8ea640 1 -- 172.21.15.32:0/2359511773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56119e3a0910 con 0x56119e39af40 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.049+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8000c00 tx=0x7f1dc800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db938457f578d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1dc2ffd640 1 -- 172.21.15.32:0/2359511773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8015040 con 0x56119e39af40 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1dc2ffd640 1 -- 172.21.15.32:0/2359511773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc8004710 con 0x56119e39af40 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1dc2ffd640 1 -- 172.21.15.32:0/2359511773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8004030 con 0x56119e39af40 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 -- 172.21.15.32:0/2359511773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 msgr2=0x56119e3a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8000c00 tx=0x7f1dc800f730 comp rx=0 tx=0).stop 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 -- 172.21.15.32:0/2359511773 shutdown_connections 2024-04-23T18:30:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e3a5800 0x56119e3bd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39af40 0x56119e3a4c00 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 --2- 172.21.15.32:0/2359511773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e39a620 0x56119e39daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 -- 172.21.15.32:0/2359511773 >> 172.21.15.32:0/2359511773 conn(0x56119e396b30 msgr2=0x56119e39bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 -- 172.21.15.32:0/2359511773 shutdown_connections 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.050+0000 7f1de206d740 1 -- 172.21.15.32:0/2359511773 wait complete. 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 -- start start 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 Processor -- start 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e3a5800 0x56119e3a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81ae390 con 0x56119e39a620 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81af800 con 0x56119e3a5800 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81aea60 con 0x56119e39af40 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1de206d740 10 client.0 _ll_register_callbacks cb 0x7f1dc800ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1ddd0eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1ddd0eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53320/0 (socket says 172.21.15.32:53320) 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.051+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/368082673 learned_addr learned my addr 172.21.15.32:0/368082673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/368082673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 msgr2=0x56119e3a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e3a5800 msgr2=0x56119e3a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e3a5800 0x56119e3a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 -- 172.21.15.32:0/368082673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56119e41a9f0 con 0x56119e39a620 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1ddd0eb640 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40eedd0 tx=0x7f1dd40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fe3d50c0e1444c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4049040 con 0x56119e39a620 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f1dd40c1110 con 0x56119e39a620 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc8004710 con 0x56119e39a620 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1dd40c13d0 con 0x56119e39a620 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 10 client.17325 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x56119e41bc90 con 0x56119e39a620 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc8004710 con 0x56119e39a620 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f1dd403bbf0 con 0x56119e39a620 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 10 client.17325 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 10 client.17325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 20 client.17325 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e41c3b0 0x56119e41e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 4 client.17325 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.052+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56119e307200 con 0x56119e41c3b0 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.053+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e41c3b0 0x56119e41e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.053+0000 7f1dc0ff9640 10 client.17325 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.053+0000 7f1ddc8ea640 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e41c3b0 0x56119e41e800 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=402317217efbdaec server_cookie=267336f1126bdf67 in_seq=0 out_seq=0 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1dc0ff9640 1 -- 172.21.15.32:0/368082673 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56119e307200 con 0x56119e41c3b0 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1dc0ff9640 10 client.17325 handle_command_reply: tid=0 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 client.17325 shutdown 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 msgr2=0x56119e3a9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40eedd0 tx=0x7f1dd40045a0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 shutdown_connections 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x56119e41c3b0 0x56119e41e800 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56119e3a5800 0x56119e3a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56119e39af40 0x56119e3a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 --2- 172.21.15.32:0/368082673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56119e39a620 0x56119e3a9280 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1dc0ff9640 0 client.17325 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 >> 172.21.15.32:0/368082673 conn(0x56119e396b30 msgr2=0x56119e3b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 shutdown_connections 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 1 -- 172.21.15.32:0/368082673 wait complete. 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.065+0000 7f1de206d740 20 client.17325 trim_cache size 0 max 16384 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:30:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 msgr2=0x7f1dc40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 0x7f1dc40a61d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009fd0 tx=0x7f1dd0009600 comp rx=0 tx=0).stop 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 msgr2=0x7f1dd81a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4005940 tx=0x7f1dd40042f0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 shutdown_connections 2024-04-23T18:30:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd810e8f0 0x7f1dd819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1dd8101660 0x7f1dd81a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1dc40a3d10 0x7f1dc40a61d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 --2- 172.21.15.32:0/3661927880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd8100d40 0x7f1dd81a06a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 >> 172.21.15.32:0/3661927880 conn(0x7f1dd8077f70 msgr2=0x7f1dd810ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 shutdown_connections 2024-04-23T18:30:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:36.066+0000 7f1ddf34f640 1 -- 172.21.15.32:0/3661927880 wait complete. 2024-04-23T18:30:37.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:37 smithi032 ceph-mon[29221]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 1.06k op/s 2024-04-23T18:30:37.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:37 smithi040 ceph-mon[32626]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 1.06k op/s 2024-04-23T18:30:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:37 smithi064 ceph-mon[33062]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 1.06k op/s 2024-04-23T18:30:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 910.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:30:39.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:39 smithi032 ceph-mon[29221]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 958 KiB/s wr, 106 op/s 2024-04-23T18:30:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:39 smithi040 ceph-mon[32626]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 958 KiB/s wr, 106 op/s 2024-04-23T18:30:39.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:39 smithi064 ceph-mon[33062]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 958 KiB/s wr, 106 op/s 2024-04-23T18:30:41.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:41 smithi040 ceph-mon[32626]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 980 KiB/s wr, 112 op/s 2024-04-23T18:30:41.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:41 smithi064 ceph-mon[33062]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 980 KiB/s wr, 112 op/s 2024-04-23T18:30:41.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:41 smithi032 ceph-mon[29221]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 980 KiB/s wr, 112 op/s 2024-04-23T18:30:43.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:43 smithi040 ceph-mon[32626]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 140 KiB/s wr, 16 op/s 2024-04-23T18:30:43.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:43 smithi064 ceph-mon[33062]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 140 KiB/s wr, 16 op/s 2024-04-23T18:30:43.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:43 smithi032 ceph-mon[29221]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 140 KiB/s wr, 16 op/s 2024-04-23T18:30:43.861 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 915.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:30:45.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:45 smithi040 ceph-mon[32626]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 140 KiB/s wr, 17 op/s 2024-04-23T18:30:45.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:45 smithi064 ceph-mon[33062]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 140 KiB/s wr, 17 op/s 2024-04-23T18:30:45.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:45 smithi032 ceph-mon[29221]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 140 KiB/s wr, 17 op/s 2024-04-23T18:30:47.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:47 smithi040 ceph-mon[32626]: pgmap v492: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 12 op/s 2024-04-23T18:30:47.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:47 smithi064 ceph-mon[33062]: pgmap v492: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 12 op/s 2024-04-23T18:30:47.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:47 smithi032 ceph-mon[29221]: pgmap v492: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 12 op/s 2024-04-23T18:30:48.864 INFO:teuthology.orchestra.run.smithi032.stderr:progress: 920.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-23T18:30:48.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/539699394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88101a40 msgr2=0x7f9c8810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/539699394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88101a40 0x7f9c8810dc70 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009ca0 tx=0x7f9c8002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/539699394 shutdown_connections 2024-04-23T18:30:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/539699394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88101a40 0x7f9c8810dc70 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/539699394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 0x7f9c88101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/539699394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88106ed0 0x7f9c881072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/539699394 >> 172.21.15.32:0/539699394 conn(0x7f9c88077c40 msgr2=0x7f9c88078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/539699394 shutdown_connections 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.109+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/539699394 wait complete. 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 Processor -- start 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 -- start start 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 0x7f9c88198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 0x7f9c88198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881110b0 con 0x7f9c88106ed0 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88110f30 con 0x7f9c88101a40 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8e6fa640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88111230 con 0x7f9c881010a0 2024-04-23T18:30:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8cc97640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8cc97640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37948/0 (socket says 172.21.15.32:37948) 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.110+0000 7f9c8cc97640 1 -- 172.21.15.32:0/4205207323 learned_addr learned my addr 172.21.15.32:0/4205207323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c87fff640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 0x7f9c88198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c877fe640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 0x7f9c88198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 -- 172.21.15.32:0/4205207323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 msgr2=0x7f9c88198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 0x7f9c88198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 -- 172.21.15.32:0/4205207323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 msgr2=0x7f9c88198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 0x7f9c88198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 -- 172.21.15.32:0/4205207323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8819d750 con 0x7f9c88106ed0 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c877fe640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 0x7f9c88198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8cc97640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9c80005cc0 tx=0x7f9c80031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c80003b70 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8819d9e0 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c800318f0 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8002f950 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.111+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c881a52d0 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.113+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c88068e00 con 0x7f9c88106ed0 2024-04-23T18:30:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.115+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9c80004b40 con 0x7f9c88106ed0 2024-04-23T18:30:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.115+0000 7f9c857fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 0x7f9c700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.115+0000 7f9c87fff640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 0x7f9c700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.115+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f9c800f1c60 con 0x7f9c88106ed0 2024-04-23T18:30:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.116+0000 7f9c87fff640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 0x7f9c700a61e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009b50 tx=0x7f9c78005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.118+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c800be430 con 0x7f9c88106ed0 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 -- 172.21.15.32:0/1096078325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8102b90 msgr2=0x7f56e8106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 --2- 172.21.15.32:0/1096078325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8102b90 0x7f56e8106bd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009900 tx=0x7f56d402a410 comp rx=0 tx=0).stop 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 -- 172.21.15.32:0/1096078325 shutdown_connections 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 --2- 172.21.15.32:0/1096078325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81072b0 0x7f56e810ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 --2- 172.21.15.32:0/1096078325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8102b90 0x7f56e8106bd0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 --2- 172.21.15.32:0/1096078325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e81021e0 0x7f56e81025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 -- 172.21.15.32:0/1096078325 >> 172.21.15.32:0/1096078325 conn(0x7f56e80fda00 msgr2=0x7f56e80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 -- 172.21.15.32:0/1096078325 shutdown_connections 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.163+0000 7f56ef260640 1 -- 172.21.15.32:0/1096078325 wait complete. 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 Processor -- start 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 -- start start 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 0x7f56e8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 0x7f56e81a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81121f0 con 0x7f56e81072b0 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8112070 con 0x7f56e8074cd0 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ef260640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8112370 con 0x7f56e81021e0 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37954/0 (socket says 172.21.15.32:37954) 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56e7fff640 1 -- 172.21.15.32:0/717566281 learned_addr learned my addr 172.21.15.32:0/717566281 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ecffc640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 0x7f56e8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.164+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 0x7f56e81a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 -- 172.21.15.32:0/717566281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 msgr2=0x7f56e8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 0x7f56e8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 -- 172.21.15.32:0/717566281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 msgr2=0x7f56e81a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 0x7f56e81a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 -- 172.21.15.32:0/717566281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e81a6fe0 con 0x7f56e81072b0 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 0x7f56e81a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56ecffc640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 0x7f56e8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e7fff640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f56d4006010 tx=0x7f56d4004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e5ffb640 1 -- 172.21.15.32:0/717566281 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d403f030 con 0x7f56e81072b0 2024-04-23T18:30:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e81a7270 con 0x7f56e81072b0 2024-04-23T18:30:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e5ffb640 1 -- 172.21.15.32:0/717566281 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d4003aa0 con 0x7f56e81072b0 2024-04-23T18:30:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56e5ffb640 1 -- 172.21.15.32:0/717566281 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4035070 con 0x7f56e81072b0 2024-04-23T18:30:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.165+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e81a77b0 con 0x7f56e81072b0 2024-04-23T18:30:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 Processor -- start 2024-04-23T18:30:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56e5ffb640 1 -- 172.21.15.32:0/717566281 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f56d4042030 con 0x7f56e81072b0 2024-04-23T18:30:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 -- start start 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9ec3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3f40 0x555cb9ece2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555cb9e28970 con 0x555cb9ece800 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cb9dc54a0 con 0x555cb9ec3f40 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56f1f72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555cb9e24930 con 0x555cb9ec3620 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37966/0 (socket says 172.21.15.32:37966) 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56e7fff640 1 -- 172.21.15.32:0/3544506627 learned_addr learned my addr 172.21.15.32:0/3544506627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9ec3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ecffc640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3f40 0x555cb9ece2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 -- 172.21.15.32:0/3544506627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3f40 msgr2=0x555cb9ece2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3f40 0x555cb9ece2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 -- 172.21.15.32:0/3544506627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 msgr2=0x555cb9ed0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56ed7fd640 1 -- 172.21.15.32:0/3544506627 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cb9b3b210 con 0x555cb9ec3620 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.168+0000 7f56e7fff640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56e5ffb640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 0x7f56d80a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9ec3a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009f10 tx=0x7f56e00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62495173ede152f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56ecffc640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 0x7f56d80a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56e5ffb640 1 -- 172.21.15.32:0/717566281 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56d40f0c90 con 0x7f56e81072b0 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56d2ffd640 1 -- 172.21.15.32:0/3544506627 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e001c680 con 0x555cb9ec3620 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56d2ffd640 1 -- 172.21.15.32:0/3544506627 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e001ccd0 con 0x555cb9ec3620 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.169+0000 7f56d2ffd640 1 -- 172.21.15.32:0/3544506627 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0011d70 con 0x555cb9ec3620 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- 172.21.15.32:0/3544506627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 msgr2=0x555cb9ec3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56ecffc640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 0x7f56d80a6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x555cb9ec4bc0 tx=0x7f56dc0084c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9ec3a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009f10 tx=0x7f56e00107a0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- 172.21.15.32:0/3544506627 shutdown_connections 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ed0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3f40 0x555cb9ece2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3544506627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9ec3a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- 172.21.15.32:0/3544506627 >> 172.21.15.32:0/3544506627 conn(0x555cb9ebfb30 msgr2=0x555cb9ebff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- 172.21.15.32:0/3544506627 shutdown_connections 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- 172.21.15.32:0/3544506627 wait complete. 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 -- start start 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 Processor -- start 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ece150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 0x555cb9ecb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.170+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56f1f72740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555cb9ed1430 con 0x555cb9ece800 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56f1f72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cb9dc30b0 con 0x555cb9ec3620 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56f1f72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555cb9ed15b0 con 0x555cb9ec3f40 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56f1f72740 10 client.0 _ll_register_callbacks cb 0x555cb9db0d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ed7fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 0x555cb9ecb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ed7fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37978/0 (socket says 172.21.15.32:37978) 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ed7fd640 1 -- 172.21.15.32:0/3543239562 learned_addr learned my addr 172.21.15.32:0/3543239562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ecffc640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ece150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 -- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 msgr2=0x555cb9ece150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ece150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 -- 172.21.15.32:0/3543239562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 msgr2=0x555cb9ecb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 -- 172.21.15.32:0/3543239562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cb9c07620 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ecffc640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ece150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56e7fff640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 0x555cb9ecb440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f56d40eea00 tx=0x7f56d403b640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a8c128fda0184f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4049070 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56d40b92c0 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.171+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56e001ccd0 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f56d40bca20 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 10 client.26851 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x555cb9ed1430 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e001ccd0 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f56d40f5040 con 0x555cb9ec3f40 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 10 client.26851 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 10 client.26851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 20 client.26851 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9ede7b0 0x555cb9ee0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 4 client.26851 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cb9e36390 con 0x555cb9ede7b0 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.172+0000 7f56ecffc640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9ede7b0 0x555cb9ee0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.173+0000 7f56c7fff640 10 client.26851 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.173+0000 7f56ecffc640 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9ede7b0 0x555cb9ee0bd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f289fa42badb33 server_cookie=a49fc0f755eee273 in_seq=0 out_seq=0 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.173+0000 7f56c7fff640 1 -- 172.21.15.32:0/3543239562 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555cb9e36390 con 0x555cb9ede7b0 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.173+0000 7f56c7fff640 10 client.26851 handle_command_reply: tid=0 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.173+0000 7f56f1f72740 1 client.26851 shutdown 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 msgr2=0x555cb9ecb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 0x555cb9ecb440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f56d40eea00 tx=0x7f56d403b640 comp rx=0 tx=0).stop 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 shutdown_connections 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9ede7b0 0x555cb9ee0bd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ece800 0x555cb9ecb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3f40 0x555cb9ecb440 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56c7fff640 0 client.26851 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 --2- 172.21.15.32:0/3543239562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ece150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 >> 172.21.15.32:0/3543239562 conn(0x555cb9ebfb30 msgr2=0x555cb9ec5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 shutdown_connections 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 1 -- 172.21.15.32:0/3543239562 wait complete. 2024-04-23T18:30:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.174+0000 7f56f1f72740 20 client.26851 trim_cache size 0 max 16384 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 Processor -- start 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 -- start start 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9eda2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 0x555cb9ee64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81b2660 con 0x555cb9ec3f40 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81afed0 con 0x555cb9ece800 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56f1f72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81add40 con 0x555cb9ec3620 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37980/0 (socket says 172.21.15.32:37980) 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 -- 172.21.15.32:0/699148692 learned_addr learned my addr 172.21.15.32:0/699148692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 0x555cb9ee64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56ecffc640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9eda2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 -- 172.21.15.32:0/699148692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 msgr2=0x555cb9eda2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9eda2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 -- 172.21.15.32:0/699148692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 msgr2=0x555cb9ee64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 0x555cb9ee64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 -- 172.21.15.32:0/699148692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cb9ee1230 con 0x555cb9ec3f40 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 0x555cb9ee64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56ecffc640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9eda2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.189+0000 7f56e7fff640 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f56d40bc950 tx=0x7f56d4003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328a7740f70d72ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56d27fc640 1 -- 172.21.15.32:0/699148692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4049070 con 0x555cb9ec3f40 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56d27fc640 1 -- 172.21.15.32:0/699148692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d40bd350 con 0x555cb9ec3f40 2024-04-23T18:30:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56d27fc640 1 -- 172.21.15.32:0/699148692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d40bc5b0 con 0x555cb9ec3f40 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 -- 172.21.15.32:0/699148692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 msgr2=0x555cb9ecdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f56d40bc950 tx=0x7f56d4003e60 comp rx=0 tx=0).stop 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 -- 172.21.15.32:0/699148692 shutdown_connections 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ece800 0x555cb9ee64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ecdc80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 --2- 172.21.15.32:0/699148692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ec3620 0x555cb9eda2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 -- 172.21.15.32:0/699148692 >> 172.21.15.32:0/699148692 conn(0x555cb9ebfb30 msgr2=0x555cb9ebecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 -- 172.21.15.32:0/699148692 shutdown_connections 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.190+0000 7f56f1f72740 1 -- 172.21.15.32:0/699148692 wait complete. 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 -- start start 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 Processor -- start 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ed6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ece800 0x555cb9ed22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81ae390 con 0x555cb9ec3f40 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81af800 con 0x555cb9ec3620 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56e81aea60 con 0x555cb9ece800 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56f1f72740 10 client.0 _ll_register_callbacks cb 0x7f56d40f44d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37988/0 (socket says 172.21.15.32:37988) 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56e7fff640 1 -- 172.21.15.32:0/2463730932 learned_addr learned my addr 172.21.15.32:0/2463730932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56ecffc640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ed6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.191+0000 7f56ed7fd640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ece800 0x555cb9ed22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 -- 172.21.15.32:0/2463730932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ece800 msgr2=0x555cb9ed22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ece800 0x555cb9ed22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.189 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 -- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 msgr2=0x555cb9ed6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ed6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 -- 172.21.15.32:0/2463730932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cb9ed2820 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56e7fff640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f56d403d040 tx=0x7f56d40dd400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4585e8159ea5c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d40d5020 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f56d40c5630 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56d40bd350 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f56d4049070 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56f1f72740 10 client.17349 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x555cb9ed2820 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d40bd350 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.192+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f56d4081bd0 con 0x555cb9ec3f40 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 10 client.17349 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 10 client.17349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 20 client.17349 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9f44c60 0x555cb9f47050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 4 client.17349 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cb9e30200 con 0x555cb9f44c60 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56ecffc640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9f44c60 0x555cb9f47050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56ecffc640 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9f44c60 0x555cb9f47050 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c1c55ec6057424 server_cookie=30e4443270e2438c in_seq=0 out_seq=0 2024-04-23T18:30:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.193+0000 7f56c7fff640 10 client.17349 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.206+0000 7f56c7fff640 1 -- 172.21.15.32:0/2463730932 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cb9e30200 con 0x555cb9f44c60 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.206+0000 7f56c7fff640 10 client.17349 handle_command_reply: tid=0 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.206+0000 7f56f1f72740 1 client.17349 shutdown 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 msgr2=0x555cb9ed1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f56d403d040 tx=0x7f56d40dd400 comp rx=0 tx=0).stop 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 shutdown_connections 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x555cb9f44c60 0x555cb9f47050 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cb9ece800 0x555cb9ed22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cb9ec3f40 0x555cb9ed1da0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56c7fff640 0 client.17349 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 --2- 172.21.15.32:0/2463730932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb9ec3620 0x555cb9ed6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 >> 172.21.15.32:0/2463730932 conn(0x555cb9ebfb30 msgr2=0x555cb9ec4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 shutdown_connections 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 1 -- 172.21.15.32:0/2463730932 wait complete. 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56f1f72740 20 client.17349 trim_cache size 0 max 16384 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:30:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 msgr2=0x7f56d80a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 0x7f56d80a6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x555cb9ec4bc0 tx=0x7f56dc0084c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 msgr2=0x7f56e8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f56d4006010 tx=0x7f56d4004060 comp rx=0 tx=0).stop 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 shutdown_connections 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56e8074cd0 0x7f56e81a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.207+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f56d80a3d40 0x7f56d80a6200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.208+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e81072b0 0x7f56e8074790 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.208+0000 7f56ef260640 1 --2- 172.21.15.32:0/717566281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e81021e0 0x7f56e8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.208+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 >> 172.21.15.32:0/717566281 conn(0x7f56e80fda00 msgr2=0x7f56e80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.208+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 shutdown_connections 2024-04-23T18:30:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:30:49.208+0000 7f56ef260640 1 -- 172.21.15.32:0/717566281 wait complete. 2024-04-23T18:30:49.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:49 smithi032 ceph-mon[29221]: pgmap v493: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-23T18:30:49.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.353+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c881a5530 con 0x7f9c88106ed0 2024-04-23T18:30:49.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.353+0000 7f9c857fa640 1 -- 172.21.15.32:0/4205207323 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11854 (secure 0 0 0) 0x7f9c800bdd30 con 0x7f9c88106ed0 2024-04-23T18:30:49.346 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:30:49.346 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:30:23.814504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:30:49.346 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 53 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 msgr2=0x7f9c700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 0x7f9c700a61e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009b50 tx=0x7f9c78005f90 comp rx=0 tx=0).stop 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 msgr2=0x7f9c8819d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9c80005cc0 tx=0x7f9c80031700 comp rx=0 tx=0).stop 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 shutdown_connections 2024-04-23T18:30:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9c700a3d20 0x7f9c700a61e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88106ed0 0x7f9c8819d020 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c88101a40 0x7f9c88198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 --2- 172.21.15.32:0/4205207323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c881010a0 0x7f9c88198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 >> 172.21.15.32:0/4205207323 conn(0x7f9c88077c40 msgr2=0x7f9c88069430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 shutdown_connections 2024-04-23T18:30:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.355+0000 7f9c8e6fa640 1 -- 172.21.15.32:0/4205207323 wait complete. 2024-04-23T18:30:49.359 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id fd2e910d --release --await' 2024-04-23T18:30:49.360 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id fd2e910d --release --await 2024-04-23T18:30:49.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:49 smithi040 ceph-mon[32626]: pgmap v493: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-23T18:30:49.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:49 smithi064 ceph-mon[33062]: pgmap v493: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-23T18:30:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- 172.21.15.32:0/3779125135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 msgr2=0x7f83a810eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- 172.21.15.32:0/3779125135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a810eb60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009ca0 tx=0x7f83a002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- 172.21.15.32:0/3779125135 shutdown_connections 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- 172.21.15.32:0/3779125135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a810eb60 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- 172.21.15.32:0/3779125135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a8103b70 0x7f83a810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- 172.21.15.32:0/3779125135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a81031c0 0x7f83a81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- 172.21.15.32:0/3779125135 >> 172.21.15.32:0/3779125135 conn(0x7f83a8077f70 msgr2=0x7f83a8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- 172.21.15.32:0/3779125135 shutdown_connections 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- 172.21.15.32:0/3779125135 wait complete. 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 Processor -- start 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- start start 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a81031c0 0x7f83a819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a8198720 0x7f83a8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81123a0 con 0x7f83a810c6d0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83a8112220 con 0x7f83a8198720 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.477+0000 7f83b0376640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83a8112520 con 0x7f83a81031c0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38012/0 (socket says 172.21.15.32:38012) 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 -- 172.21.15.32:0/4108015132 learned_addr learned my addr 172.21.15.32:0/4108015132 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ae112640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a81031c0 0x7f83a819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ae913640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a8198720 0x7f83a8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 -- 172.21.15.32:0/4108015132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a81031c0 msgr2=0x7f83a819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a81031c0 0x7f83a819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 -- 172.21.15.32:0/4108015132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a8198720 msgr2=0x7f83a8198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a8198720 0x7f83a8198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 -- 172.21.15.32:0/4108015132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a8199320 con 0x7f83a810c6d0 2024-04-23T18:30:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83ad911640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f839800d750 tx=0x7f839800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f839f7fe640 1 -- 172.21.15.32:0/4108015132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398017070 con 0x7f83a810c6d0 2024-04-23T18:30:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f839f7fe640 1 -- 172.21.15.32:0/4108015132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83980054d0 con 0x7f83a810c6d0 2024-04-23T18:30:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f839f7fe640 1 -- 172.21.15.32:0/4108015132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398004c30 con 0x7f83a810c6d0 2024-04-23T18:30:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a81a5400 con 0x7f83a810c6d0 2024-04-23T18:30:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.478+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a81a5800 con 0x7f83a810c6d0 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f839f7fe640 1 -- 172.21.15.32:0/4108015132 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8398004dd0 con 0x7f83a810c6d0 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 Processor -- start 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 -- start start 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 0x561a77718f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a776736d0 con 0x561a77719480 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a776101a0 con 0x561a7770e490 2024-04-23T18:30:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83b316a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a7766f690 con 0x561a7770edb0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.481+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38028/0 (socket says 172.21.15.32:38028) 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 -- 172.21.15.32:0/2741660479 learned_addr learned my addr 172.21.15.32:0/2741660479 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f839f7fe640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 0x7f83940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ae112640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 0x561a77718f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ae913640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ae913640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 0x7f83940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f839f7fe640 1 -- 172.21.15.32:0/4108015132 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f83980d00d0 con 0x7f83a810c6d0 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 -- 172.21.15.32:0/2741660479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 msgr2=0x561a77718f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 0x561a77718f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 -- 172.21.15.32:0/2741660479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 msgr2=0x561a7770e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 -- 172.21.15.32:0/2741660479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a7761a5b0 con 0x561a77719480 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ae112640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 0x561a77718f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ad911640 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f839809c020 tx=0x7f8398009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5d4ff67ee8034a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f83ae913640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 0x7f83940a6260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f83a0004af0 tx=0x7f83a00095c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f839cbfe640 1 -- 172.21.15.32:0/2741660479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398020050 con 0x561a77719480 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f839cbfe640 1 -- 172.21.15.32:0/2741660479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f839809c790 con 0x561a77719480 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.482+0000 7f839cbfe640 1 -- 172.21.15.32:0/2741660479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398098700 con 0x561a77719480 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- 172.21.15.32:0/2741660479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 msgr2=0x561a7771b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f839809c020 tx=0x7f8398009f90 comp rx=0 tx=0).stop 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- 172.21.15.32:0/2741660479 shutdown_connections 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771b870 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770edb0 0x561a77718f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 --2- 172.21.15.32:0/2741660479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- 172.21.15.32:0/2741660479 >> 172.21.15.32:0/2741660479 conn(0x561a7770a7b0 msgr2=0x561a7770ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- 172.21.15.32:0/2741660479 shutdown_connections 2024-04-23T18:30:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- 172.21.15.32:0/2741660479 wait complete. 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.483+0000 7f83b316a740 1 -- start start 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 Processor -- start 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a77718ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77716900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a7771c250 con 0x561a7770edb0 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561a7760db20 con 0x561a7770e490 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a7771c3d0 con 0x561a77719480 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83b316a740 10 client.0 _ll_register_callbacks cb 0x561a7760d530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38032/0 (socket says 172.21.15.32:38032) 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 -- 172.21.15.32:0/3413605485 learned_addr learned my addr 172.21.15.32:0/3413605485 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ae112640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a77718ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ae913640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77716900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 -- 172.21.15.32:0/3413605485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 msgr2=0x561a77716900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77716900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 -- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 msgr2=0x561a77718ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a77718ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 -- 172.21.15.32:0/3413605485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a77385920 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ae913640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77716900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f83ad911640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f83980cde40 tx=0x7f83980cdd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eb4702475934b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.484+0000 7f8391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398098330 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f8398098700 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f839809c790 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f83980a0440 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 10 client.17361 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561a77451d30 con 0x561a7770edb0 2024-04-23T18:30:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f839809c790 con 0x561a7770edb0 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f83980a4870 con 0x561a7770edb0 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 10 client.17361 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 10 client.17361 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 20 client.17361 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a7772b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 4 client.17361 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a77631500 con 0x561a77729420 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.485+0000 7f83ae112640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a7772b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.486+0000 7f83ae112640 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a7772b810 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10778d71eff35db server_cookie=d6933455da588f73 in_seq=0 out_seq=0 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.486+0000 7f8391ffb640 10 client.17361 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f8391ffb640 1 -- 172.21.15.32:0/3413605485 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a77631500 con 0x561a77729420 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f8391ffb640 10 client.17361 handle_command_reply: tid=0 2024-04-23T18:30:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 client.17361 shutdown 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 msgr2=0x561a777161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f83980cde40 tx=0x7f83980cdd70 comp rx=0 tx=0).stop 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 shutdown_connections 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a7772b810 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77716900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f8391ffb640 0 client.17361 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a777161c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 --2- 172.21.15.32:0/3413605485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a77718ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 >> 172.21.15.32:0/3413605485 conn(0x561a7770a7b0 msgr2=0x561a777105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 shutdown_connections 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 1 -- 172.21.15.32:0/3413605485 wait complete. 2024-04-23T18:30:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.487+0000 7f83b316a740 20 client.17361 trim_cache size 0 max 16384 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 Processor -- start 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 -- start start 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77718f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81b0470 con 0x561a7770edb0 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81adce0 con 0x561a7770e490 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83b316a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81abb50 con 0x561a77719480 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.504+0000 7f83ad911640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38042/0 (socket says 172.21.15.32:38042) 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 -- 172.21.15.32:0/2304983754 learned_addr learned my addr 172.21.15.32:0/2304983754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ae913640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77718f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 -- 172.21.15.32:0/2304983754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 msgr2=0x561a77718f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77718f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ae112640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 -- 172.21.15.32:0/2304983754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 msgr2=0x561a7770a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 -- 172.21.15.32:0/2304983754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a77715170 con 0x561a7770edb0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ae913640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77718f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f83ad911640 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f83980057c0 tx=0x7f83980cdcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0376c474c70a51d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f8393fff640 1 -- 172.21.15.32:0/2304983754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398020050 con 0x561a7770edb0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f8393fff640 1 -- 172.21.15.32:0/2304983754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83980b9020 con 0x561a7770edb0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.505+0000 7f8393fff640 1 -- 172.21.15.32:0/2304983754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839809a1c0 con 0x561a7770edb0 2024-04-23T18:30:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- 172.21.15.32:0/2304983754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 msgr2=0x561a77718a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f83980057c0 tx=0x7f83980cdcd0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- 172.21.15.32:0/2304983754 shutdown_connections 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a77719480 0x561a77718f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a7770edb0 0x561a77718a00 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- 172.21.15.32:0/2304983754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770e490 0x561a7770a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- 172.21.15.32:0/2304983754 >> 172.21.15.32:0/2304983754 conn(0x561a7770a7b0 msgr2=0x561a77711480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- 172.21.15.32:0/2304983754 shutdown_connections 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- 172.21.15.32:0/2304983754 wait complete. 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- start start 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 Processor -- start 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770e490 0x561a7771a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 0x561a7771a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81ac1a0 con 0x561a77719480 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81ad610 con 0x561a7770edb0 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81ac870 con 0x561a7770e490 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83b316a740 10 client.0 _ll_register_callbacks cb 0x7f8398009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83ae913640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83ae913640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38050/0 (socket says 172.21.15.32:38050) 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.506+0000 7f83ae913640 1 -- 172.21.15.32:0/3830742896 learned_addr learned my addr 172.21.15.32:0/3830742896 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae112640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770e490 0x561a7771a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ad911640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 0x561a7771a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 -- 172.21.15.32:0/3830742896 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770e490 msgr2=0x561a7771a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770e490 0x561a7771a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 -- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 msgr2=0x561a7771a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 0x561a7771a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 -- 172.21.15.32:0/3830742896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a7770ebe0 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ad911640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 0x561a7771a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83ae913640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009ca0 tx=0x7f83a002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6842bd6f4a3c4246 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f8391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0004250 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f83a00043f0 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83980b9020 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f83a00045f0 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83b316a740 10 client.17367 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.507+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561a7778f890 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83980b9020 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f83a0055070 con 0x561a77719480 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 10 client.17367 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 10 client.17367 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 20 client.17367 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a777920b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 4 client.17367 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fd2e910d", "release": true, "await": true}] 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fd2e910d", "release": true, "await": true}) -- 0x561a776810f0 con 0x561a77729420 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83ae112640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a777920b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f83ae112640 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a777920b0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c721d5ee6043fd8b server_cookie=f923d4975a28cbfc in_seq=0 out_seq=0 2024-04-23T18:30:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.508+0000 7f8391ffb640 10 client.17367 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f8391ffb640 1 -- 172.21.15.32:0/3830742896 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561a776810f0 con 0x561a77729420 2024-04-23T18:30:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f8391ffb640 10 client.17367 handle_command_reply: tid=0 2024-04-23T18:30:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 client.17367 shutdown 2024-04-23T18:30:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 msgr2=0x561a7771ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009ca0 tx=0x7f83a002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 shutdown_connections 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561a77729420 0x561a777920b0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a77719480 0x561a7771ae50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a7770edb0 0x561a7771a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f8391ffb640 0 client.17367 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 --2- 172.21.15.32:0/3830742896 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a7770e490 0x561a7771a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 >> 172.21.15.32:0/3830742896 conn(0x561a7770a7b0 msgr2=0x561a7770fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 shutdown_connections 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 1 -- 172.21.15.32:0/3830742896 wait complete. 2024-04-23T18:30:49.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.517+0000 7f83b316a740 20 client.17367 trim_cache size 0 max 16384 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 53, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 52, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "fd2e910d": { 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 52, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:30:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 84.8, 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RELEASED", 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:49.513 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 msgr2=0x7f83940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 0x7f83940a6260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f83a0004af0 tx=0x7f83a00095c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 msgr2=0x7f83a819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f839800d750 tx=0x7f839800dc10 comp rx=0 tx=0).stop 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 shutdown_connections 2024-04-23T18:30:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83a8198720 0x7f83a8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f83940a3da0 0x7f83940a6260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83a810c6d0 0x7f83a819eba0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 --2- 172.21.15.32:0/4108015132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83a81031c0 0x7f83a819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 >> 172.21.15.32:0/4108015132 conn(0x7f83a8077f70 msgr2=0x7f83a80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 shutdown_connections 2024-04-23T18:30:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.518+0000 7f83b0376640 1 -- 172.21.15.32:0/4108015132 wait complete. 2024-04-23T18:30:49.524 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fd2e910d', seconds elapsed: 26.5 2024-04-23T18:30:49.524 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.85 2024-04-23T18:30:49.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/3641916558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 msgr2=0x7f2cf810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/3641916558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf810dd30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009ca0 tx=0x7f2cf002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/3641916558 shutdown_connections 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/3641916558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf810dd30 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/3641916558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101a40 0x7f2cf8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/3641916558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf8101090 0x7f2cf8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/3641916558 >> 172.21.15.32:0/3641916558 conn(0x7f2cf8078060 msgr2=0x7f2cf8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/3641916558 shutdown_connections 2024-04-23T18:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.690+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/3641916558 wait complete. 2024-04-23T18:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 Processor -- start 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 -- start start 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 0x7f2cf8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf819d000 0x7f2cf819d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf81110a0 con 0x7f2cf8106160 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8110f20 con 0x7f2cf8101090 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cfd7f7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8111220 con 0x7f2cf819d000 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cf67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cf6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 0x7f2cf8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cf67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38064/0 (socket says 172.21.15.32:38064) 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.691+0000 7f2cf67fc640 1 -- 172.21.15.32:0/4039280564 learned_addr learned my addr 172.21.15.32:0/4039280564 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf77fe640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf819d000 0x7f2cf819d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 -- 172.21.15.32:0/4039280564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf819d000 msgr2=0x7f2cf819d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf819d000 0x7f2cf819d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 -- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 msgr2=0x7f2cf8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 -- 172.21.15.32:0/4039280564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf819daf0 con 0x7f2cf8101090 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf67fc640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cf6ffd640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 0x7f2cf8198740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009ce0 tx=0x7f2ce800b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8004020 con 0x7f2cf8101090 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf819dde0 con 0x7f2cf8101090 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce80041c0 con 0x7f2cf8101090 2024-04-23T18:30:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8019670 con 0x7f2cf8101090 2024-04-23T18:30:49.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.692+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf8102dc0 con 0x7f2cf8101090 2024-04-23T18:30:49.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.693+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf8102560 con 0x7f2cf8101090 2024-04-23T18:30:49.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.694+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2ce80035f0 con 0x7f2cf8101090 2024-04-23T18:30:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.695+0000 7f2cdffff640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 0x7f2ce00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.695+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f2ce80d4090 con 0x7f2cf8101090 2024-04-23T18:30:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.695+0000 7f2cf67fc640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 0x7f2ce00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.696+0000 7f2cf67fc640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 0x7f2ce00a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0095f0 tx=0x7f2cec005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:49.701+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ce80a00e0 con 0x7f2cf8101090 2024-04-23T18:30:49.707 INFO:teuthology.orchestra.run.smithi032.stdout:transaction type: 2024-04-23T18:30:49.707 INFO:teuthology.orchestra.run.smithi032.stdout:scaling factor: 32 2024-04-23T18:30:49.707 INFO:teuthology.orchestra.run.smithi032.stdout:query mode: simple 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:number of clients: 100 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:number of threads: 4 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:duration: 900 s 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:number of transactions actually processed: 1565783 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:latency average = 58.795 ms 2024-04-23T18:30:49.708 INFO:teuthology.orchestra.run.smithi032.stdout:latency stddev = 791.190 ms 2024-04-23T18:30:49.709 INFO:teuthology.orchestra.run.smithi032.stdout:tps = 1700.370927 (including connections establishing) 2024-04-23T18:30:49.709 INFO:teuthology.orchestra.run.smithi032.stdout:tps = 1700.375964 (excluding connections establishing) 2024-04-23T18:30:49.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-23T18:30:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.079+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cf8069250 con 0x7f2cf8101090 2024-04-23T18:30:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.081+0000 7f2cdffff640 1 -- 172.21.15.32:0/4039280564 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11854 (secure 0 0 0) 0x7f2ce80a43e0 con 0x7f2cf8101090 2024-04-23T18:30:50.074 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:30:50.075 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:30:23.814504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:30:50.075 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 53 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 msgr2=0x7f2ce00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 0x7f2ce00a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0095f0 tx=0x7f2cec005f90 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 msgr2=0x7f2cf8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 0x7f2cf8198740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009ce0 tx=0x7f2ce800b690 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 shutdown_connections 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cf819d000 0x7f2cf819d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f2ce00a3d20 0x7f2ce00a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cf8106160 0x7f2cf8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 --2- 172.21.15.32:0/4039280564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8101090 0x7f2cf8198740 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 >> 172.21.15.32:0/4039280564 conn(0x7f2cf8078060 msgr2=0x7f2cf8104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 shutdown_connections 2024-04-23T18:30:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.089+0000 7f2cfd7f7640 1 -- 172.21.15.32:0/4039280564 wait complete. 2024-04-23T18:30:50.101 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db / --timeout 60 --expiration 84.85 --await-for 42.0' 2024-04-23T18:30:50.102 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db / --timeout 60 --expiration 84.85 --await-for 42.0 2024-04-23T18:30:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.323+0000 7faa8e02a640 1 -- 172.21.15.32:0/3799148324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa88110770 msgr2=0x7faa88108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.323+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3799148324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa88110770 0x7faa88108f50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009900 tx=0x7faa7c02a440 comp rx=0 tx=0).stop 2024-04-23T18:30:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.325+0000 7faa8e02a640 1 -- 172.21.15.32:0/3799148324 shutdown_connections 2024-04-23T18:30:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.325+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3799148324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8811b730 0x7faa8811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.325+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3799148324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa88109490 0x7faa88109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.325+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3799148324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa88110770 0x7faa88108f50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.325+0000 7faa8e02a640 1 -- 172.21.15.32:0/3799148324 >> 172.21.15.32:0/3799148324 conn(0x7faa8806e2e0 msgr2=0x7faa8806e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:50.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.330+0000 7faa8e02a640 1 -- 172.21.15.32:0/3799148324 shutdown_connections 2024-04-23T18:30:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.334+0000 7faa8e02a640 1 -- 172.21.15.32:0/3799148324 wait complete. 2024-04-23T18:30:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.334+0000 7faa8e02a640 1 Processor -- start 2024-04-23T18:30:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.335+0000 7faa8e02a640 1 -- start start 2024-04-23T18:30:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.335+0000 7faa8e02a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa88109490 0x7faa88119020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.335+0000 7faa8e02a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa88110770 0x7faa88110fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.336+0000 7faa8e02a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.336+0000 7faa8e02a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa88121160 con 0x7faa88109490 2024-04-23T18:30:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.336+0000 7faa8e02a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa88120fe0 con 0x7faa8811b730 2024-04-23T18:30:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.336+0000 7faa8e02a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa881212e0 con 0x7faa88110770 2024-04-23T18:30:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.340+0000 7faa87fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.340+0000 7faa87fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49336/0 (socket says 172.21.15.32:49336) 2024-04-23T18:30:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.340+0000 7faa87fff640 1 -- 172.21.15.32:0/3960531029 learned_addr learned my addr 172.21.15.32:0/3960531029 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.341+0000 7faa87fff640 1 -- 172.21.15.32:0/3960531029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa88110770 msgr2=0x7faa88110fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.341+0000 7faa87fff640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa88110770 0x7faa88110fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.341+0000 7faa87fff640 1 -- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa88109490 msgr2=0x7faa88119020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.341+0000 7faa87fff640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa88109490 0x7faa88119020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.341+0000 7faa87fff640 1 -- 172.21.15.32:0/3960531029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa88111d40 con 0x7faa8811b730 2024-04-23T18:30:50.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.345+0000 7faa87fff640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faa8000b9e0 tx=0x7faa8000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.345+0000 7faa84ff9640 1 -- 172.21.15.32:0/3960531029 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80019070 con 0x7faa8811b730 2024-04-23T18:30:50.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.346+0000 7faa84ff9640 1 -- 172.21.15.32:0/3960531029 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa80014440 con 0x7faa8811b730 2024-04-23T18:30:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.346+0000 7faa84ff9640 1 -- 172.21.15.32:0/3960531029 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa800125a0 con 0x7faa8811b730 2024-04-23T18:30:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.346+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa881bc4e0 con 0x7faa8811b730 2024-04-23T18:30:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.346+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa881bc980 con 0x7faa8811b730 2024-04-23T18:30:50.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.353+0000 7faa84ff9640 1 -- 172.21.15.32:0/3960531029 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7faa8001c070 con 0x7faa8811b730 2024-04-23T18:30:50.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.354+0000 7faa84ff9640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 0x7faa700a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.355+0000 7faa84ff9640 1 -- 172.21.15.32:0/3960531029 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7faa800cfb40 con 0x7faa8811b730 2024-04-23T18:30:50.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.356+0000 7faa877fe640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 0x7faa700a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.357+0000 7faa90e1e740 1 Processor -- start 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 -- start start 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c0a0 0x561cec28c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec299480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561cec1f1370 con 0x561cec28c9c0 2024-04-23T18:30:50.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cec18e2e0 con 0x561cec28c0a0 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.360+0000 7faa90e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561cec1ed330 con 0x561cec297090 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38086/0 (socket says 172.21.15.32:38086) 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 -- 172.21.15.32:0/2036738215 learned_addr learned my addr 172.21.15.32:0/2036738215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 -- 172.21.15.32:0/2036738215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 msgr2=0x561cec299480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:30:50.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec299480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 -- 172.21.15.32:0/2036738215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c0a0 msgr2=0x561cec28c480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:30:50.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c0a0 0x561cec28c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.361+0000 7faa86ffd640 1 -- 172.21.15.32:0/2036738215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cebfcfd80 con 0x561cec28c9c0 2024-04-23T18:30:50.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.362+0000 7faa86ffd640 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7faa78009bb0 tx=0x7faa7800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9adad110bc78c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.364+0000 7faa6dffb640 1 -- 172.21.15.32:0/2036738215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78005020 con 0x561cec28c9c0 2024-04-23T18:30:50.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.364+0000 7faa6dffb640 1 -- 172.21.15.32:0/2036738215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa780047a0 con 0x561cec28c9c0 2024-04-23T18:30:50.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.365+0000 7faa90e1e740 1 -- 172.21.15.32:0/2036738215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 msgr2=0x561cec296b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.365+0000 7faa90e1e740 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7faa78009bb0 tx=0x7faa7800e720 comp rx=0 tx=0).stop 2024-04-23T18:30:50.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.367+0000 7faa877fe640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 0x7faa700a63b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0096c0 tx=0x7faa7c000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:50 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4205207323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.372+0000 7faa90e1e740 1 -- 172.21.15.32:0/2036738215 shutdown_connections 2024-04-23T18:30:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.372+0000 7faa90e1e740 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec299480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.372+0000 7faa90e1e740 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c9c0 0x561cec296b50 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.372+0000 7faa90e1e740 1 --2- 172.21.15.32:0/2036738215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c0a0 0x561cec28c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.372+0000 7faa90e1e740 1 -- 172.21.15.32:0/2036738215 >> 172.21.15.32:0/2036738215 conn(0x561cec2883c0 msgr2=0x561cec2887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:50.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.382+0000 7faa90e1e740 1 -- 172.21.15.32:0/2036738215 shutdown_connections 2024-04-23T18:30:50.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.385+0000 7faa90e1e740 1 -- 172.21.15.32:0/2036738215 wait complete. 2024-04-23T18:30:50.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.385+0000 7faa90e1e740 1 -- start start 2024-04-23T18:30:50.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 Processor -- start 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c9c0 0x561cec2944c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec294a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561cec299ed0 con 0x561cec28c0a0 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cec18bfb0 con 0x561cec28c9c0 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561cec29a050 con 0x561cec297090 2024-04-23T18:30:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.386+0000 7faa90e1e740 10 client.0 _ll_register_callbacks cb 0x561cec18b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.388+0000 7faa877fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.388+0000 7faa877fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38098/0 (socket says 172.21.15.32:38098) 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.388+0000 7faa877fe640 1 -- 172.21.15.32:0/3566543449 learned_addr learned my addr 172.21.15.32:0/3566543449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.390+0000 7faa877fe640 1 -- 172.21.15.32:0/3566543449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 msgr2=0x561cec294a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.390+0000 7faa877fe640 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec294a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.390+0000 7faa877fe640 1 -- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c9c0 msgr2=0x561cec2944c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.390+0000 7faa877fe640 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c9c0 0x561cec2944c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.390+0000 7faa877fe640 1 -- 172.21.15.32:0/3566543449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cec030510 con 0x561cec28c0a0 2024-04-23T18:30:50.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.394+0000 7faa877fe640 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faa7c004590 tx=0x7faa7c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c48fc0aa64bd94 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.394+0000 7faa66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:30:50.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.395+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c04b020 con 0x561cec28c0a0 2024-04-23T18:30:50.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.395+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7faa7c003e30 con 0x561cec28c0a0 2024-04-23T18:30:50.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.395+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa780047a0 con 0x561cec28c0a0 2024-04-23T18:30:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.400+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7faa7c035040 con 0x561cec28c0a0 2024-04-23T18:30:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.400+0000 7faa90e1e740 10 client.17382 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.400+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561cebec3670 con 0x561cec28c0a0 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa780047a0 con 0x561cec28c0a0 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7faa7c0315f0 con 0x561cec28c0a0 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 10 client.17382 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 10 client.17382 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 20 client.17382 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec2a75a0 0x561cec2a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 4 client.17382 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.405+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cec1fed90 con 0x561cec2a75a0 2024-04-23T18:30:50.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.410+0000 7faa86ffd640 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec2a75a0 0x561cec2a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.416+0000 7faa86ffd640 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec2a75a0 0x561cec2a9a00 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc40df551d0c1d82 server_cookie=5ab41631df6b6417 in_seq=0 out_seq=0 2024-04-23T18:30:50.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.416+0000 7faa66ffd640 10 client.17382 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.417+0000 7faa66ffd640 1 -- 172.21.15.32:0/3566543449 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561cec1fed90 con 0x561cec2a75a0 2024-04-23T18:30:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.417+0000 7faa66ffd640 10 client.17382 handle_command_reply: tid=0 2024-04-23T18:30:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.417+0000 7faa90e1e740 1 client.17382 shutdown 2024-04-23T18:30:50.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.420+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 msgr2=0x561cec293f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.420+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faa7c004590 tx=0x7faa7c009260 comp rx=0 tx=0).stop 2024-04-23T18:30:50.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 shutdown_connections 2024-04-23T18:30:50.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec2a75a0 0x561cec2a9a00 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec297090 0x561cec294a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec28c9c0 0x561cec2944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3566543449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec28c0a0 0x561cec293f80 secure :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faa7c004590 tx=0x7faa7c009260 comp rx=0 tx=0).stop 2024-04-23T18:30:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 >> 172.21.15.32:0/3566543449 conn(0x561cec2883c0 msgr2=0x561cec287600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.422+0000 7faa66ffd640 0 client.17382 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:50.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:50 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4205207323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:50.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.431+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 shutdown_connections 2024-04-23T18:30:50.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.431+0000 7faa90e1e740 1 -- 172.21.15.32:0/3566543449 wait complete. 2024-04-23T18:30:50.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.435+0000 7faa90e1e740 20 client.17382 trim_cache size 0 max 16384 2024-04-23T18:30:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.476+0000 7faa90e1e740 1 Processor -- start 2024-04-23T18:30:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.476+0000 7faa90e1e740 1 -- start start 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec295e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec2a2e00 0x561cec294850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38104/0 (socket says 172.21.15.32:38104) 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa86ffd640 1 -- 172.21.15.32:0/4041153572 learned_addr learned my addr 172.21.15.32:0/4041153572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa74011170 con 0x561cec297090 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7400e9e0 con 0x561cec2a2e00 2024-04-23T18:30:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.477+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa7400c850 con 0x561cec28c9c0 2024-04-23T18:30:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.478+0000 7faa87fff640 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec2a2e00 0x561cec294850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.478+0000 7faa877fe640 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec295e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.479+0000 7faa86ffd640 1 -- 172.21.15.32:0/4041153572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 msgr2=0x561cec295e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.479+0000 7faa86ffd640 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec295e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.479+0000 7faa86ffd640 1 -- 172.21.15.32:0/4041153572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec2a2e00 msgr2=0x561cec294850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.479+0000 7faa86ffd640 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec2a2e00 0x561cec294850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.479+0000 7faa86ffd640 1 -- 172.21.15.32:0/4041153572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cec28c7f0 con 0x561cec297090 2024-04-23T18:30:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.484+0000 7faa86ffd640 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7faa78009b80 tx=0x7faa78009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8031fa92a1adb4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.486+0000 7faa6d7fa640 1 -- 172.21.15.32:0/4041153572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7802c020 con 0x561cec297090 2024-04-23T18:30:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.486+0000 7faa6d7fa640 1 -- 172.21.15.32:0/4041153572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa78026070 con 0x561cec297090 2024-04-23T18:30:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.487+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 msgr2=0x561cec287d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.487+0000 7faa90e1e740 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7faa78009b80 tx=0x7faa78009990 comp rx=0 tx=0).stop 2024-04-23T18:30:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.491+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 shutdown_connections 2024-04-23T18:30:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.491+0000 7faa90e1e740 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec2a2e00 0x561cec294850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.491+0000 7faa90e1e740 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec297090 0x561cec287d80 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.491+0000 7faa90e1e740 1 --2- 172.21.15.32:0/4041153572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec295e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.491+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 >> 172.21.15.32:0/4041153572 conn(0x561cec2883c0 msgr2=0x561cec2a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.492+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 shutdown_connections 2024-04-23T18:30:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.496+0000 7faa90e1e740 1 -- 172.21.15.32:0/4041153572 wait complete. 2024-04-23T18:30:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.496+0000 7faa90e1e740 1 -- start start 2024-04-23T18:30:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.496+0000 7faa90e1e740 1 Processor -- start 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec29f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec2a2e00 0x561cec29c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa7400cea0 con 0x561cec2a2e00 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7400e310 con 0x561cec297090 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa7400d570 con 0x561cec28c9c0 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa90e1e740 10 client.0 _ll_register_callbacks cb 0x7faa7c004240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.32:49370/0 (socket says 172.21.15.32:49370) 2024-04-23T18:30:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.497+0000 7faa86ffd640 1 -- 172.21.15.32:0/3695785552 learned_addr learned my addr 172.21.15.32:0/3695785552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.500+0000 7faa86ffd640 1 -- 172.21.15.32:0/3695785552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 msgr2=0x561cec29f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.500+0000 7faa86ffd640 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec29f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.500+0000 7faa86ffd640 1 -- 172.21.15.32:0/3695785552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec2a2e00 msgr2=0x561cec29c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.500+0000 7faa86ffd640 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec2a2e00 0x561cec29c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.500+0000 7faa86ffd640 1 -- 172.21.15.32:0/3695785552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cec292f50 con 0x561cec297090 2024-04-23T18:30:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.501+0000 7faa86ffd640 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faa78009ff0 tx=0x7faa78005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da054d6f3c2be0db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.502+0000 7faa66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-23T18:30:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.506+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78035060 con 0x561cec297090 2024-04-23T18:30:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.506+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7faa78009ca0 con 0x561cec297090 2024-04-23T18:30:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.506+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa78026070 con 0x561cec297090 2024-04-23T18:30:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.510+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7faa7802c050 con 0x561cec297090 2024-04-23T18:30:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.511+0000 7faa90e1e740 10 client.26855 fetch_fsmap learned FSMap version 53 2024-04-23T18:30:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.511+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561cec30e770 con 0x561cec297090 2024-04-23T18:30:50.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:50 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4205207323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa78026070 con 0x561cec297090 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7faa78038050 con 0x561cec297090 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 10 client.26855 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 10 client.26855 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 20 client.26855 populate_metadata read hostname 'smithi032' 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec30ecc0 0x561cec3110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:50.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 4 client.26855 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.85, "await_for": 42.0}] 2024-04-23T18:30:50.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.517+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.85, "await_for": 42.0}) -- 0x561cec1f8c00 con 0x561cec30ecc0 2024-04-23T18:30:50.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.519+0000 7faa877fe640 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec30ecc0 0x561cec3110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:50.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.530+0000 7faa877fe640 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec30ecc0 0x561cec3110e0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=860881b118d56531 server_cookie=546faaed8cc0901 in_seq=0 out_seq=0 2024-04-23T18:30:50.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:50.530+0000 7faa66ffd640 10 client.26855 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:51.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:51 smithi040 ceph-mon[32626]: pgmap v494: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-23T18:30:51.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:51 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/4039280564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:51.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.393+0000 7faa66ffd640 1 -- 172.21.15.32:0/3695785552 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561cec1f8c00 con 0x561cec30ecc0 2024-04-23T18:30:51.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.393+0000 7faa66ffd640 10 client.26855 handle_command_reply: tid=0 2024-04-23T18:30:51.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.393+0000 7faa90e1e740 1 client.26855 shutdown 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 msgr2=0x561cec29c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faa78009ff0 tx=0x7faa78005b00 comp rx=0 tx=0).stop 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 shutdown_connections 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x561cec30ecc0 0x561cec3110e0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561cec2a2e00 0x561cec29c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cec297090 0x561cec29c350 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 --2- 172.21.15.32:0/3695785552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561cec28c9c0 0x561cec29f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 >> 172.21.15.32:0/3695785552 conn(0x561cec2883c0 msgr2=0x561cec286d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa66ffd640 0 client.26855 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:30:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 shutdown_connections 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.394+0000 7faa90e1e740 1 -- 172.21.15.32:0/3695785552 wait complete. 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa90e1e740 20 client.26855 trim_cache size 0 max 16384 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 53, 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:30:51.388 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 54, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "23f38b46": { 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 54, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.0, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 84.8, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:30:51.389 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:30:51.390 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 msgr2=0x7faa700a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 0x7faa700a63b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0096c0 tx=0x7faa7c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 msgr2=0x7faa88111520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faa8000b9e0 tx=0x7faa8000bea0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 shutdown_connections 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8811b730 0x7faa88111520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa88110770 0x7faa88110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa700a3ef0 0x7faa700a63b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 --2- 172.21.15.32:0/3960531029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa88109490 0x7faa88119020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.395+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 >> 172.21.15.32:0/3960531029 conn(0x7faa8806e2e0 msgr2=0x7faa88119720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:51.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.396+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 shutdown_connections 2024-04-23T18:30:51.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:30:51.396+0000 7faa8e02a640 1 -- 172.21.15.32:0/3960531029 wait complete. 2024-04-23T18:30:51.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:51 smithi032 ceph-mon[29221]: pgmap v494: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-23T18:30:51.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:51 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/4039280564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:51.407 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '23f38b46', quiesce took 1.9 seconds. Will release after: 24.85 2024-04-23T18:30:51.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:51 smithi064 ceph-mon[33062]: pgmap v494: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-23T18:30:51.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:51 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/4039280564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:53.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:53 smithi040 ceph-mon[32626]: pgmap v495: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-04-23T18:30:53.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:53 smithi064 ceph-mon[33062]: pgmap v495: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-04-23T18:30:53.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:53 smithi032 ceph-mon[29221]: pgmap v495: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-04-23T18:30:55.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:55 smithi040 ceph-mon[32626]: pgmap v496: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 891 KiB/s wr, 99 op/s 2024-04-23T18:30:55.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:55 smithi064 ceph-mon[33062]: pgmap v496: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 891 KiB/s wr, 99 op/s 2024-04-23T18:30:55.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:55 smithi032 ceph-mon[29221]: pgmap v496: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 891 KiB/s wr, 99 op/s 2024-04-23T18:30:57.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:57 smithi040 ceph-mon[32626]: pgmap v497: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-23T18:30:57.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:57 smithi064 ceph-mon[33062]: pgmap v497: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-23T18:30:57.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:57 smithi032 ceph-mon[29221]: pgmap v497: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-23T18:30:59.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:30:59 smithi040 ceph-mon[32626]: pgmap v498: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-04-23T18:30:59.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:30:59 smithi064 ceph-mon[33062]: pgmap v498: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-04-23T18:30:59.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:30:59 smithi032 ceph-mon[29221]: pgmap v498: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-04-23T18:31:01.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:01 smithi040 ceph-mon[32626]: pgmap v499: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.9 MiB/s wr, 959 op/s 2024-04-23T18:31:01.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:01 smithi064 ceph-mon[33062]: pgmap v499: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.9 MiB/s wr, 959 op/s 2024-04-23T18:31:01.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:01 smithi032 ceph-mon[29221]: pgmap v499: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.9 MiB/s wr, 959 op/s 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 -- 172.21.15.32:0/3734988704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad81031e0 msgr2=0x7f5ad81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 --2- 172.21.15.32:0/3734988704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad81031e0 0x7f5ad81035c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009900 tx=0x7f5ac802a480 comp rx=0 tx=0).stop 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 -- 172.21.15.32:0/3734988704 shutdown_connections 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 --2- 172.21.15.32:0/3734988704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 0x7f5ad810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 --2- 172.21.15.32:0/3734988704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad8103b00 0x7f5ad810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 --2- 172.21.15.32:0/3734988704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad81031e0 0x7f5ad81035c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 -- 172.21.15.32:0/3734988704 >> 172.21.15.32:0/3734988704 conn(0x7f5ad8077f70 msgr2=0x7f5ad8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 -- 172.21.15.32:0/3734988704 shutdown_connections 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 -- 172.21.15.32:0/3734988704 wait complete. 2024-04-23T18:31:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.308+0000 7f5add0e8640 1 Processor -- start 2024-04-23T18:31:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 -- start start 2024-04-23T18:31:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad81031e0 0x7f5ad8100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 0x7f5ad81018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad8112200 con 0x7f5ad8103b00 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad8112080 con 0x7f5ad810c5b0 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5add0e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad8112380 con 0x7f5ad81031e0 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56324/0 (socket says 172.21.15.32:56324) 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 -- 172.21.15.32:0/1747866206 learned_addr learned my addr 172.21.15.32:0/1747866206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5ad759e640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 0x7f5ad81018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad81031e0 0x7f5ad8100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 -- 172.21.15.32:0/1747866206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad81031e0 msgr2=0x7f5ad8100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad81031e0 0x7f5ad8100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 -- 172.21.15.32:0/1747866206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 msgr2=0x7f5ad81018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 0x7f5ad81018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.309+0000 7f5acffff640 1 -- 172.21.15.32:0/1747866206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad81a6c80 con 0x7f5ad8103b00 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5acffff640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000d770 tx=0x7f5ac000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5ad4d99640 1 -- 172.21.15.32:0/1747866206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0017070 con 0x7f5ad8103b00 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5ad4d99640 1 -- 172.21.15.32:0/1747866206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac00054d0 con 0x7f5ad8103b00 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad81a6f70 con 0x7f5ad8103b00 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5ad4d99640 1 -- 172.21.15.32:0/1747866206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0004c30 con 0x7f5ad8103b00 2024-04-23T18:31:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.310+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad81a73e0 con 0x7f5ad8103b00 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.312+0000 7f5ad4d99640 1 -- 172.21.15.32:0/1747866206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5ac0004e10 con 0x7f5ad8103b00 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.312+0000 7f5adfe04740 1 Processor -- start 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 -- start start 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 0x55a85425da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a8542682c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a8541c2970 con 0x55a854268800 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a85415f4a0 con 0x55a85425d620 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5adfe04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a8541be930 con 0x55a85425df40 2024-04-23T18:31:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56340/0 (socket says 172.21.15.32:56340) 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5ad4d99640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5acffff640 1 -- 172.21.15.32:0/312001035 learned_addr learned my addr 172.21.15.32:0/312001035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a8542682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.313+0000 7f5ad759e640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 0x55a85425da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5ad4d99640 1 -- 172.21.15.32:0/1747866206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f5ac00d02e0 con 0x7f5ad8103b00 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5ad759e640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 -- 172.21.15.32:0/312001035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 msgr2=0x55a8542682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a8542682c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 -- 172.21.15.32:0/312001035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 msgr2=0x55a85425da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 0x55a85425da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 -- 172.21.15.32:0/312001035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a853ed5210 con 0x55a854268800 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5ad759e640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 0x55a85425da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a8542682c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acffff640 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009610 tx=0x7f5ac0005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4815f546145db2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5ad759e640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009fd0 tx=0x7f5ad0004ab0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acd7fa640 1 -- 172.21.15.32:0/312001035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac009c240 con 0x55a854268800 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acd7fa640 1 -- 172.21.15.32:0/312001035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac00a05f0 con 0x55a854268800 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.314+0000 7f5acd7fa640 1 -- 172.21.15.32:0/312001035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a4a20 con 0x55a854268800 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- 172.21.15.32:0/312001035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 msgr2=0x55a85426abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009610 tx=0x7f5ac0005680 comp rx=0 tx=0).stop 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- 172.21.15.32:0/312001035 shutdown_connections 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a854268800 0x55a85426abf0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a8542682c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 --2- 172.21.15.32:0/312001035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85425d620 0x55a85425da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- 172.21.15.32:0/312001035 >> 172.21.15.32:0/312001035 conn(0x55a854259b30 msgr2=0x55a854259f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- 172.21.15.32:0/312001035 shutdown_connections 2024-04-23T18:31:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- 172.21.15.32:0/312001035 wait complete. 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.315+0000 7f5adfe04740 1 -- start start 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 Processor -- start 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425d620 0x55a854268130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854265960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a85426b370 con 0x55a85425df40 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a85415d0b0 con 0x55a854268800 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a85426b4f0 con 0x55a85425d620 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5adfe04740 10 client.0 _ll_register_callbacks cb 0x55a85414ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56348/0 (socket says 172.21.15.32:56348) 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 -- 172.21.15.32:0/3682664677 learned_addr learned my addr 172.21.15.32:0/3682664677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5ad759e640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854265960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425d620 0x55a854268130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 -- 172.21.15.32:0/3682664677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425d620 msgr2=0x55a854268130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425d620 0x55a854268130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 -- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 msgr2=0x55a854265960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854265960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 -- 172.21.15.32:0/3682664677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a853fa1620 con 0x55a85425df40 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5acffff640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0098c10 tx=0x7f5ac00a4eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccf948f8d0700d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.316+0000 7f5abe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00b9020 con 0x55a85425df40 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f5ac009a540 con 0x55a85425df40 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac00a05f0 con 0x55a85425df40 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5ac00988b0 con 0x55a85425df40 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 10 client.17397 fetch_fsmap learned FSMap version 53 2024-04-23T18:31:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a8542772c0 con 0x55a85425df40 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac00a05f0 con 0x55a85425df40 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f5ac00b0ce0 con 0x55a85425df40 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 10 client.17397 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 10 client.17397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 20 client.17397 populate_metadata read hostname 'smithi032' 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a854278610 0x55a85427aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 4 client.17397 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.317+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8541d0390 con 0x55a854278610 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.318+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a854278610 0x55a85427aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.318+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a854278610 0x55a85427aa00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b1aca118b25bb03 server_cookie=ade0f4ac606686 in_seq=0 out_seq=0 2024-04-23T18:31:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.318+0000 7f5abe7fc640 10 client.17397 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:31:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5abe7fc640 1 -- 172.21.15.32:0/3682664677 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8541d0390 con 0x55a854278610 2024-04-23T18:31:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5abe7fc640 10 client.17397 handle_command_reply: tid=0 2024-04-23T18:31:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 client.17397 shutdown 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 msgr2=0x55a854265420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0098c10 tx=0x7f5ac00a4eb0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 shutdown_connections 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a854278610 0x55a85427aa00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854265960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425df40 0x55a854265420 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 --2- 172.21.15.32:0/3682664677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425d620 0x55a854268130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5abe7fc640 0 client.17397 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 >> 172.21.15.32:0/3682664677 conn(0x55a854259b30 msgr2=0x55a85425f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 shutdown_connections 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.319+0000 7f5adfe04740 1 -- 172.21.15.32:0/3682664677 wait complete. 2024-04-23T18:31:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.320+0000 7f5adfe04740 20 client.17397 trim_cache size 0 max 16384 2024-04-23T18:31:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 Processor -- start 2024-04-23T18:31:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 -- start start 2024-04-23T18:31:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854267c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854280030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81b2660 con 0x55a85425d620 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81afed0 con 0x55a854268800 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5adfe04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81add40 con 0x55a85425df40 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56350/0 (socket says 172.21.15.32:56350) 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/180528330 learned_addr learned my addr 172.21.15.32:0/180528330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad759e640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854280030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5acffff640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854267c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/180528330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 msgr2=0x55a854267c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854267c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/180528330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 msgr2=0x55a854280030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.335+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854280030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/180528330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a854265cc0 con 0x55a85425d620 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5acffff640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854267c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8031440 tx=0x7f5ac8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75c81d1d72fcc92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5accff9640 1 -- 172.21.15.32:0/180528330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8049070 con 0x55a85425d620 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5accff9640 1 -- 172.21.15.32:0/180528330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac80044d0 con 0x55a85425d620 2024-04-23T18:31:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5accff9640 1 -- 172.21.15.32:0/180528330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8035040 con 0x55a85425d620 2024-04-23T18:31:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.336+0000 7f5adfe04740 1 -- 172.21.15.32:0/180528330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 msgr2=0x55a854274230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8031440 tx=0x7f5ac8005c00 comp rx=0 tx=0).stop 2024-04-23T18:31:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- 172.21.15.32:0/180528330 shutdown_connections 2024-04-23T18:31:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854280030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854267c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- 172.21.15.32:0/180528330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854274230 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- 172.21.15.32:0/180528330 >> 172.21.15.32:0/180528330 conn(0x55a854259b30 msgr2=0x55a854258cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- 172.21.15.32:0/180528330 shutdown_connections 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- 172.21.15.32:0/180528330 wait complete. 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- start start 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 Processor -- start 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854261cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854262230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81ae390 con 0x55a85425d620 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81af800 con 0x55a854268800 2024-04-23T18:31:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad81aea60 con 0x55a85425df40 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5adfe04740 10 client.0 _ll_register_callbacks cb 0x7f5ac00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.337+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56360/0 (socket says 172.21.15.32:56360) 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/428156621 learned_addr learned my addr 172.21.15.32:0/428156621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5acffff640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854261cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad759e640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854262230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/428156621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 msgr2=0x55a854261cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854261cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 msgr2=0x55a854262230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854262230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 -- 172.21.15.32:0/428156621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a854262970 con 0x55a85425d620 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad759e640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854262230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5acffff640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854261cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5ad6d9d640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009230 tx=0x7f5ac80442e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9d635ebb58b36a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5abe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8035050 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7f5ac8005450 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac80044d0 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5ac8049070 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5adfe04740 10 client.17403 fetch_fsmap learned FSMap version 53 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.338+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a8542de800 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac80044d0 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f5ac80037b0 con 0x55a85425d620 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 10 client.17403 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 10 client.17403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 20 client.17403 populate_metadata read hostname 'smithi032' 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a8542deee0 0x55a8542e1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 4 client.17403 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8541ca200 con 0x55a8542deee0 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5acffff640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a8542deee0 0x55a8542e1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5acffff640 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a8542deee0 0x55a8542e1300 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cd68b921f72bc45 server_cookie=962eade24c7b9091 in_seq=0 out_seq=0 2024-04-23T18:31:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:31:02.339+0000 7f5abe7fc640 10 client.17403 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:31:03.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:03 smithi040 ceph-mon[32626]: pgmap v500: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.9 MiB/s wr, 954 op/s 2024-04-23T18:31:03.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:03 smithi064 ceph-mon[33062]: pgmap v500: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.9 MiB/s wr, 954 op/s 2024-04-23T18:31:03.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:03 smithi032 ceph-mon[29221]: pgmap v500: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.9 MiB/s wr, 954 op/s 2024-04-23T18:31:05.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:05 smithi040 ceph-mon[32626]: pgmap v501: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.9 MiB/s wr, 955 op/s 2024-04-23T18:31:05.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:05 smithi064 ceph-mon[33062]: pgmap v501: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.9 MiB/s wr, 955 op/s 2024-04-23T18:31:05.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:05 smithi032 ceph-mon[29221]: pgmap v501: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.9 MiB/s wr, 955 op/s 2024-04-23T18:31:07.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:07 smithi040 ceph-mon[32626]: pgmap v502: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.1 MiB/s wr, 887 op/s 2024-04-23T18:31:07.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:07 smithi040 ceph-mon[32626]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:31:07.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:07 smithi064 ceph-mon[33062]: pgmap v502: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.1 MiB/s wr, 887 op/s 2024-04-23T18:31:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:07 smithi064 ceph-mon[33062]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:31:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:07 smithi032 ceph-mon[29221]: pgmap v502: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.1 MiB/s wr, 887 op/s 2024-04-23T18:31:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:07 smithi032 ceph-mon[29221]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:31:09.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:09 smithi040 ceph-mon[32626]: pgmap v504: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.3 MiB/s wr, 469 op/s 2024-04-23T18:31:09.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:09 smithi064 ceph-mon[33062]: pgmap v504: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.3 MiB/s wr, 469 op/s 2024-04-23T18:31:09.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:09 smithi032 ceph-mon[29221]: pgmap v504: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 4.3 MiB/s wr, 469 op/s 2024-04-23T18:31:11.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:11 smithi040 ceph-mon[32626]: pgmap v505: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 105 KiB/s wr, 32 op/s 2024-04-23T18:31:11.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:11 smithi040 ceph-mon[32626]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:31:11.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:11 smithi064 ceph-mon[33062]: pgmap v505: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 105 KiB/s wr, 32 op/s 2024-04-23T18:31:11.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:11 smithi064 ceph-mon[33062]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:31:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:11 smithi032 ceph-mon[29221]: pgmap v505: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 105 KiB/s wr, 32 op/s 2024-04-23T18:31:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:11 smithi032 ceph-mon[29221]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:31:13.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:13 smithi040 ceph-mon[32626]: pgmap v507: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 130 KiB/s wr, 38 op/s 2024-04-23T18:31:13.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:13 smithi064 ceph-mon[33062]: pgmap v507: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 130 KiB/s wr, 38 op/s 2024-04-23T18:31:13.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:13 smithi032 ceph-mon[29221]: pgmap v507: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 130 KiB/s wr, 38 op/s 2024-04-23T18:31:15.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:15 smithi040 ceph-mon[32626]: pgmap v508: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T18:31:15.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:15 smithi064 ceph-mon[33062]: pgmap v508: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T18:31:15.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:15 smithi032 ceph-mon[29221]: pgmap v508: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T18:31:16.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- 172.21.15.32:0/987141697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 msgr2=0x7f05a010ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 --2- 172.21.15.32:0/987141697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a010ede0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0598009ca0 tx=0x7f059802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- 172.21.15.32:0/987141697 shutdown_connections 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 --2- 172.21.15.32:0/987141697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a010ede0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 --2- 172.21.15.32:0/987141697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0069250 0x7f05a00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 --2- 172.21.15.32:0/987141697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0108040 0x7f05a0108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- 172.21.15.32:0/987141697 >> 172.21.15.32:0/987141697 conn(0x7f05a0077c40 msgr2=0x7f05a0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- 172.21.15.32:0/987141697 shutdown_connections 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- 172.21.15.32:0/987141697 wait complete. 2024-04-23T18:31:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 Processor -- start 2024-04-23T18:31:16.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.379+0000 7f05a69e5640 1 -- start start 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0069250 0x7f05a01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 0x7f05a019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f05a0112180 con 0x7f05a0069bf0 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05a0112000 con 0x7f05a0069250 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a69e5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f05a0112300 con 0x7f05a0108040 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48010/0 (socket says 172.21.15.32:48010) 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 -- 172.21.15.32:0/1031232241 learned_addr learned my addr 172.21.15.32:0/1031232241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059ffff640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0069250 0x7f05a01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a4f82640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 0x7f05a019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 -- 172.21.15.32:0/1031232241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 msgr2=0x7f05a019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 0x7f05a019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 -- 172.21.15.32:0/1031232241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0069250 msgr2=0x7f05a01a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0069250 0x7f05a01a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 -- 172.21.15.32:0/1031232241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a019af20 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f05a4f82640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 0x7f05a019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.380+0000 7f059f7fe640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f059000d950 tx=0x7f059000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.381+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590017070 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.381+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0590009c30 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.381+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059000f620 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.381+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a019b210 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.381+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a01a7520 con 0x7f05a0069bf0 2024-04-23T18:31:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.383+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a0103650 con 0x7f05a0069bf0 2024-04-23T18:31:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.384+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0590010460 con 0x7f05a0069bf0 2024-04-23T18:31:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.384+0000 7f059d7fa640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 0x7f058c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.384+0000 7f059ffff640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 0x7f058c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.384+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f05900d04f0 con 0x7f05a0069bf0 2024-04-23T18:31:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.384+0000 7f059ffff640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 0x7f058c0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0588005fd0 tx=0x7f0588005850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.389+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f059009cbb0 con 0x7f05a0069bf0 2024-04-23T18:31:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.616+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a01a76f0 con 0x7f05a0069bf0 2024-04-23T18:31:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.617+0000 7f059d7fa640 1 -- 172.21.15.32:0/1031232241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11854 (secure 0 0 0) 0x7f0590009de0 con 0x7f05a0069bf0 2024-04-23T18:31:16.610 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:31:16.610 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:30:23.814504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:31:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 53 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 msgr2=0x7f058c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 0x7f058c0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0588005fd0 tx=0x7f0588005850 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 msgr2=0x7f05a01a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f059000d950 tx=0x7f059000de10 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 shutdown_connections 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05a0108040 0x7f05a019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f058c0a3d20 0x7f058c0a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05a0069bf0 0x7f05a01a0ae0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 --2- 172.21.15.32:0/1031232241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05a0069250 0x7f05a01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 >> 172.21.15.32:0/1031232241 conn(0x7f05a0077c40 msgr2=0x7f05a00ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:16.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 shutdown_connections 2024-04-23T18:31:16.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.619+0000 7f05a69e5640 1 -- 172.21.15.32:0/1031232241 wait complete. 2024-04-23T18:31:16.623 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --set-id 23f38b46 --release --await' 2024-04-23T18:31:16.623 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --set-id 23f38b46 --release --await 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2546456319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4102a60 msgr2=0x7fc6f4102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 --2- 172.21.15.32:0/2546456319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4102a60 0x7fc6f4102ec0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009930 tx=0x7fc6e802a420 comp rx=0 tx=0).stop 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2546456319 shutdown_connections 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 --2- 172.21.15.32:0/2546456319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f4103400 0x7fc6f410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 --2- 172.21.15.32:0/2546456319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4102a60 0x7fc6f4102ec0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 --2- 172.21.15.32:0/2546456319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6f4101180 0x7fc6f4101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2546456319 >> 172.21.15.32:0/2546456319 conn(0x7fc6f4077f70 msgr2=0x7fc6f4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2546456319 shutdown_connections 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.739+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2546456319 wait complete. 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 Processor -- start 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 -- start start 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6f4101180 0x7fc6f419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f4102a60 0x7fc6f419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4103400 0x7fc6f4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41111a0 con 0x7fc6f4103400 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f4111020 con 0x7fc6f4102a60 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f9b19640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f4111320 con 0x7fc6f4101180 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4103400 0x7fc6f4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4103400 0x7fc6f4198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58074/0 (socket says 172.21.15.32:58074) 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 -- 172.21.15.32:0/2376743303 learned_addr learned my addr 172.21.15.32:0/2376743303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6f4101180 0x7fc6f419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f4102a60 0x7fc6f419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 -- 172.21.15.32:0/2376743303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6f4101180 msgr2=0x7fc6f419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6f4101180 0x7fc6f419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 -- 172.21.15.32:0/2376743303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f4102a60 msgr2=0x7fc6f419e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.740+0000 7fc6f3fff640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f4102a60 0x7fc6f419e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f3fff640 1 -- 172.21.15.32:0/2376743303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f4198dd0 con 0x7fc6f4103400 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f3fff640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f4103400 0x7fc6f4198650 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009fd0 tx=0x7fc6ec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f0ff9640 1 -- 172.21.15.32:0/2376743303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec017070 con 0x7fc6f4103400 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f0ff9640 1 -- 172.21.15.32:0/2376743303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6ec003e20 con 0x7fc6f4103400 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2376743303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f41990c0 con 0x7fc6f4103400 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f0ff9640 1 -- 172.21.15.32:0/2376743303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec010460 con 0x7fc6f4103400 2024-04-23T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.741+0000 7fc6f9b19640 1 -- 172.21.15.32:0/2376743303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f41a5310 con 0x7fc6f4103400 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f0ff9640 1 -- 172.21.15.32:0/2376743303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc6ec003970 con 0x7fc6f4103400 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 Processor -- start 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 -- start start 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64d490 0x557a3b64d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557a3b5b26d0 con 0x557a3b64ddb0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a3b54f1a0 con 0x557a3b658480 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557a3b5ae690 con 0x557a3b64d490 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58084/0 (socket says 172.21.15.32:58084) 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f37fe640 1 -- 172.21.15.32:0/3630495725 learned_addr learned my addr 172.21.15.32:0/3630495725 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f0ff9640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc6dc0a3d20 0x7fc6dc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.744+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64d490 0x557a3b64d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f3fff640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc6dc0a3d20 0x7fc6dc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 -- 172.21.15.32:0/3630495725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64d490 msgr2=0x557a3b64d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64d490 0x557a3b64d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 -- 172.21.15.32:0/3630495725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 msgr2=0x557a3b65a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 -- 172.21.15.32:0/3630495725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a3b5595b0 con 0x557a3b64ddb0 2024-04-23T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f0ff9640 1 -- 172.21.15.32:0/2376743303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7fc6ec0d0500 con 0x7fc6f4103400 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009d20 tx=0x7fc6e4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51df9f6d993a78bb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6f3fff640 1 --2- 172.21.15.32:0/2376743303 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fc6dc0a3d20 0x7fc6dc0a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec09c270 tx=0x7fc6ec098810 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6d9ffb640 1 -- 172.21.15.32:0/3630495725 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e401e070 con 0x557a3b64ddb0 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6d9ffb640 1 -- 172.21.15.32:0/3630495725 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6e4019ae0 con 0x557a3b64ddb0 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.745+0000 7fc6d9ffb640 1 -- 172.21.15.32:0/3630495725 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e400eac0 con 0x557a3b64ddb0 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- 172.21.15.32:0/3630495725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 msgr2=0x557a3b657f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009d20 tx=0x7fc6e4011810 comp rx=0 tx=0).stop 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- 172.21.15.32:0/3630495725 shutdown_connections 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b657f40 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/3630495725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64d490 0x557a3b64d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- 172.21.15.32:0/3630495725 >> 172.21.15.32:0/3630495725 conn(0x557a3b6497b0 msgr2=0x557a3b649b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- 172.21.15.32:0/3630495725 shutdown_connections 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- 172.21.15.32:0/3630495725 wait complete. 2024-04-23T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 -- start start 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.746+0000 7fc6fc82e740 1 Processor -- start 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 0x557a3b657e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b658480 0x557a3b655740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557a3b65b310 con 0x557a3b64ddb0 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a3b54cb20 con 0x557a3b64d490 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557a3b65b490 con 0x557a3b658480 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 10 client.0 _ll_register_callbacks cb 0x557a3b54c530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58088/0 (socket says 172.21.15.32:58088) 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2038073402 learned_addr learned my addr 172.21.15.32:0/2038073402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 0x557a3b657e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b658480 0x557a3b655740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2038073402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b658480 msgr2=0x557a3b655740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b658480 0x557a3b655740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 msgr2=0x557a3b657e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 0x557a3b657e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2038073402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a3b2c4920 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 0x557a3b657e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400e760 tx=0x7fc6e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cd5bb2ef4b8e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4027020 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7fc6e4019630 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.747+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6e4019ae0 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc6e4032050 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 10 client.17421 fetch_fsmap learned FSMap version 53 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x557a3b390d30 con 0x557a3b64ddb0 2024-04-23T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6e4019ae0 con 0x557a3b64ddb0 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fc6e400ebf0 con 0x557a3b64ddb0 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 10 client.17421 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 10 client.17421 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 20 client.17421 populate_metadata read hostname 'smithi032' 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6685c0 0x557a3b66a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 4 client.17421 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a3b570500 con 0x557a3b6685c0 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.748+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6685c0 0x557a3b66a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.749+0000 7fc6d2ffd640 10 client.17421 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:31:16.749+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6685c0 0x557a3b66a9b0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=402a081b8e4db2dd server_cookie=c79abd2661887c37 in_seq=0 out_seq=0 2024-04-23T18:31:17.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:17 smithi040 ceph-mon[32626]: pgmap v509: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-23T18:31:17.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:17 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1031232241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:17 smithi064 ceph-mon[33062]: pgmap v509: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-23T18:31:17.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:17 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1031232241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:17.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:17 smithi032 ceph-mon[29221]: pgmap v509: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-23T18:31:17.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:17 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1031232241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:19.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:19 smithi040 ceph-mon[32626]: pgmap v510: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 7 op/s 2024-04-23T18:31:19.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:19 smithi064 ceph-mon[33062]: pgmap v510: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 7 op/s 2024-04-23T18:31:19.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:19 smithi032 ceph-mon[29221]: pgmap v510: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 7 op/s 2024-04-23T18:31:21.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:21 smithi040 ceph-mon[32626]: pgmap v511: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-23T18:31:21.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:21 smithi064 ceph-mon[33062]: pgmap v511: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-23T18:31:21.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:21 smithi032 ceph-mon[29221]: pgmap v511: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-23T18:31:23.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:23 smithi040 ceph-mon[32626]: pgmap v512: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-04-23T18:31:23.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:23 smithi064 ceph-mon[33062]: pgmap v512: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-04-23T18:31:23.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:23 smithi032 ceph-mon[29221]: pgmap v512: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-04-23T18:31:25.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:25 smithi064 ceph-mon[33062]: pgmap v513: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2024-04-23T18:31:25.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:25 smithi032 ceph-mon[29221]: pgmap v513: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2024-04-23T18:31:25.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:25 smithi040 ceph-mon[32626]: pgmap v513: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2024-04-23T18:31:27.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:27 smithi064 ceph-mon[33062]: pgmap v514: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-23T18:31:27.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:27 smithi032 ceph-mon[29221]: pgmap v514: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-23T18:31:27.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:27 smithi040 ceph-mon[32626]: pgmap v514: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-23T18:31:29.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:29 smithi064 ceph-mon[33062]: pgmap v515: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.7 KiB/s wr, 6 op/s 2024-04-23T18:31:29.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:29 smithi032 ceph-mon[29221]: pgmap v515: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.7 KiB/s wr, 6 op/s 2024-04-23T18:31:29.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:29 smithi040 ceph-mon[32626]: pgmap v515: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.7 KiB/s wr, 6 op/s 2024-04-23T18:31:31.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:31 smithi064 ceph-mon[33062]: pgmap v516: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:31.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:31 smithi032 ceph-mon[29221]: pgmap v516: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:31.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:31 smithi040 ceph-mon[32626]: pgmap v516: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:33.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:33 smithi064 ceph-mon[33062]: pgmap v517: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-04-23T18:31:33.637 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:33 smithi040 ceph-mon[32626]: pgmap v517: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-04-23T18:31:33.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:33 smithi032 ceph-mon[29221]: pgmap v517: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-04-23T18:31:34.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:34 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:34.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:34 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:34.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:34 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:34.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:34 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:31:34.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:34 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:34.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:34 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:34.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:34 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:34.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:34 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:31:34.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:34 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:34.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:34 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:34.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:34 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:34.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:34 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:31:35.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:35 smithi064 ceph-mon[33062]: pgmap v518: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 21 KiB/s wr, 8 op/s 2024-04-23T18:31:35.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:35 smithi032 ceph-mon[29221]: pgmap v518: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 21 KiB/s wr, 8 op/s 2024-04-23T18:31:35.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:35 smithi040 ceph-mon[32626]: pgmap v518: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 21 KiB/s wr, 8 op/s 2024-04-23T18:31:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:37 smithi064 ceph-mon[33062]: pgmap v519: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:37.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:37 smithi032 ceph-mon[29221]: pgmap v519: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:37.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:37 smithi040 ceph-mon[32626]: pgmap v519: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:31:39.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:39 smithi064 ceph-mon[33062]: pgmap v520: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:31:39.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:39 smithi032 ceph-mon[29221]: pgmap v520: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:31:39.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:39 smithi040 ceph-mon[32626]: pgmap v520: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-23T18:31:41.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:41 smithi064 ceph-mon[33062]: pgmap v521: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-23T18:31:41.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:41 smithi032 ceph-mon[29221]: pgmap v521: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-23T18:31:41.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:41 smithi040 ceph-mon[32626]: pgmap v521: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-23T18:31:43.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:43 smithi064 ceph-mon[33062]: pgmap v522: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.1 KiB/s wr, 5 op/s 2024-04-23T18:31:43.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:43 smithi032 ceph-mon[29221]: pgmap v522: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.1 KiB/s wr, 5 op/s 2024-04-23T18:31:43.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:43 smithi040 ceph-mon[32626]: pgmap v522: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.1 KiB/s wr, 5 op/s 2024-04-23T18:31:45.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:45 smithi064 ceph-mon[33062]: pgmap v523: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2024-04-23T18:31:45.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:45 smithi032 ceph-mon[29221]: pgmap v523: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2024-04-23T18:31:45.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:45 smithi040 ceph-mon[32626]: pgmap v523: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2024-04-23T18:31:47.417 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:47 smithi040 ceph-mon[32626]: pgmap v524: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-04-23T18:31:47.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:47 smithi064 ceph-mon[33062]: pgmap v524: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-04-23T18:31:47.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:47 smithi032 ceph-mon[29221]: pgmap v524: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-04-23T18:31:49.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:49 smithi064 ceph-mon[33062]: pgmap v525: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-23T18:31:49.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:49 smithi032 ceph-mon[29221]: pgmap v525: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-23T18:31:49.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:49 smithi040 ceph-mon[32626]: pgmap v525: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-23T18:31:51.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:51 smithi064 ceph-mon[33062]: pgmap v526: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:31:51.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:51 smithi032 ceph-mon[29221]: pgmap v526: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:31:51.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:51 smithi040 ceph-mon[32626]: pgmap v526: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:31:53.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:53 smithi064 ceph-mon[33062]: pgmap v527: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:31:53.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:53 smithi032 ceph-mon[29221]: pgmap v527: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:31:53.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:53 smithi040 ceph-mon[32626]: pgmap v527: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:31:55.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:55 smithi064 ceph-mon[33062]: pgmap v528: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:31:55.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:55 smithi032 ceph-mon[29221]: pgmap v528: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:31:55.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:55 smithi040 ceph-mon[32626]: pgmap v528: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:31:57.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:57 smithi064 ceph-mon[33062]: pgmap v529: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-23T18:31:57.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:57 smithi032 ceph-mon[29221]: pgmap v529: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-23T18:31:57.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:57 smithi040 ceph-mon[32626]: pgmap v529: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-23T18:31:59.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:31:59 smithi032 ceph-mon[29221]: pgmap v530: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 6 op/s 2024-04-23T18:31:59.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:31:59 smithi040 ceph-mon[32626]: pgmap v530: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 6 op/s 2024-04-23T18:31:59.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:31:59 smithi064 ceph-mon[33062]: pgmap v530: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 6 op/s 2024-04-23T18:32:01.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:01 smithi032 ceph-mon[29221]: pgmap v531: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:32:01.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:01 smithi040 ceph-mon[32626]: pgmap v531: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:32:01.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:01 smithi064 ceph-mon[33062]: pgmap v531: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:32:03.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:03 smithi032 ceph-mon[29221]: pgmap v532: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2024-04-23T18:32:03.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:03 smithi040 ceph-mon[32626]: pgmap v532: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2024-04-23T18:32:03.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:03 smithi064 ceph-mon[33062]: pgmap v532: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2024-04-23T18:32:05.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:05 smithi032 ceph-mon[29221]: pgmap v533: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2024-04-23T18:32:05.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:05 smithi040 ceph-mon[32626]: pgmap v533: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2024-04-23T18:32:05.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:05 smithi064 ceph-mon[33062]: pgmap v533: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2024-04-23T18:32:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:07 smithi032 ceph-mon[29221]: pgmap v534: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-23T18:32:07.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:07 smithi032 ceph-mon[29221]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:32:07.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:07 smithi040 ceph-mon[32626]: pgmap v534: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-23T18:32:07.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:07 smithi040 ceph-mon[32626]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:32:07.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:07 smithi064 ceph-mon[33062]: pgmap v534: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-23T18:32:07.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:07 smithi064 ceph-mon[33062]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:32:09.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:09 smithi032 ceph-mon[29221]: pgmap v536: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-23T18:32:09.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:09 smithi040 ceph-mon[32626]: pgmap v536: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-23T18:32:09.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:09 smithi064 ceph-mon[33062]: pgmap v536: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-23T18:32:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:11 smithi032 ceph-mon[29221]: pgmap v537: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-23T18:32:11.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:11 smithi032 ceph-mon[29221]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:32:11.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:11 smithi040 ceph-mon[32626]: pgmap v537: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-23T18:32:11.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:11 smithi040 ceph-mon[32626]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:32:11.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:11 smithi064 ceph-mon[33062]: pgmap v537: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-23T18:32:11.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:11 smithi064 ceph-mon[33062]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:32:13.472 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:13 smithi064 ceph-mon[33062]: pgmap v539: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 33 op/s 2024-04-23T18:32:13.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:13 smithi032 ceph-mon[29221]: pgmap v539: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 33 op/s 2024-04-23T18:32:13.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:13 smithi040 ceph-mon[32626]: pgmap v539: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 33 op/s 2024-04-23T18:32:15.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:15 smithi032 ceph-mon[29221]: pgmap v540: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2024-04-23T18:32:15.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:15 smithi040 ceph-mon[32626]: pgmap v540: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2024-04-23T18:32:15.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:15 smithi064 ceph-mon[33062]: pgmap v540: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5abe7fc640 1 -- 172.21.15.32:0/428156621 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5ac00a7070 con 0x55a8542deee0 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5abe7fc640 10 client.17403 handle_command_reply: tid=0 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 client.17403 shutdown 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 msgr2=0x55a854270a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009230 tx=0x7f5ac80442e0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 shutdown_connections 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x55a8542deee0 0x55a8542e1300 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a854268800 0x55a854262230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a85425df40 0x55a854261cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 --2- 172.21.15.32:0/428156621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a85425d620 0x55a854270a10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5abe7fc640 0 client.17403 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 >> 172.21.15.32:0/428156621 conn(0x55a854259b30 msgr2=0x55a854264cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.296+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2038073402 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a3b570500 con 0x557a3b6685c0 2024-04-23T18:32:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.296+0000 7fc6d2ffd640 10 client.17421 handle_command_reply: tid=0 2024-04-23T18:32:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.296+0000 7fc6fc82e740 1 client.17421 shutdown 2024-04-23T18:32:16.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 shutdown_connections 2024-04-23T18:32:16.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 1 -- 172.21.15.32:0/428156621 wait complete. 2024-04-23T18:32:16.291 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.296+0000 7f5adfe04740 20 client.17403 trim_cache size 0 max 16384 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.296+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 msgr2=0x557a3b655130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.296+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400e760 tx=0x7fc6e4004270 comp rx=0 tx=0).stop 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 shutdown_connections 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6685c0 0x557a3b66a9b0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b658480 0x557a3b655740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64ddb0 0x557a3b655130 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6d2ffd640 0 client.17421 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2038073402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b64d490 0x557a3b657e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 >> 172.21.15.32:0/2038073402 conn(0x557a3b6497b0 msgr2=0x557a3b64f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-23T18:32:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-23T18:32:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-23T18:32:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-23T18:32:16.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 shutdown_connections 2024-04-23T18:32:16.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2038073402 wait complete. 2024-04-23T18:32:16.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.297+0000 7fc6fc82e740 20 client.17421 trim_cache size 0 max 16384 2024-04-23T18:32:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 msgr2=0x7f5ac40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009fd0 tx=0x7f5ad0004ab0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 msgr2=0x7f5ad8101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000d770 tx=0x7f5ac000dc30 comp rx=0 tx=0).stop 2024-04-23T18:32:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 shutdown_connections 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad810c5b0 0x7f5ad81018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ad8103b00 0x7f5ad8101370 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 --2- 172.21.15.32:0/1747866206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ad81031e0 0x7f5ad8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 >> 172.21.15.32:0/1747866206 conn(0x7f5ad8077f70 msgr2=0x7f5ad80fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 shutdown_connections 2024-04-23T18:32:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-23T18:32:16.297+0000 7f5add0e8640 1 -- 172.21.15.32:0/1747866206 wait complete. 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.317+0000 7fc6fc82e740 1 Processor -- start 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.317+0000 7fc6fc82e740 1 -- start start 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b657970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b648850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41b02f0 con 0x557a3b64d490 2024-04-23T18:32:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41adb60 con 0x557a3b658480 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41ab9d0 con 0x557a3b64ddb0 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b657970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34992/0 (socket says 172.21.15.32:34992) 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f2ffd640 1 -- 172.21.15.32:0/2761879210 learned_addr learned my addr 172.21.15.32:0/2761879210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b648850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2761879210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 msgr2=0x557a3b648850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b648850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2761879210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 msgr2=0x557a3b650610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2761879210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a3b653680 con 0x557a3b64ddb0 2024-04-23T18:32:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:16.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.318+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b657970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4011850 tx=0x7fc6e4038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98adc1398c1caf46 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.319+0000 7fc6d97fa640 1 -- 172.21.15.32:0/2761879210 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4027070 con 0x557a3b64ddb0 2024-04-23T18:32:16.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.319+0000 7fc6d97fa640 1 -- 172.21.15.32:0/2761879210 <== mon.1 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6e401f040 con 0x557a3b64ddb0 2024-04-23T18:32:16.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.319+0000 7fc6d97fa640 1 -- 172.21.15.32:0/2761879210 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e401e070 con 0x557a3b64ddb0 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.319+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2761879210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 msgr2=0x557a3b657970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.319+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b657970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4011850 tx=0x7fc6e4038000 comp rx=0 tx=0).stop 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2761879210 shutdown_connections 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b648850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b657970 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2761879210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b650610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2761879210 >> 172.21.15.32:0/2761879210 conn(0x557a3b6497b0 msgr2=0x557a3b648140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2761879210 shutdown_connections 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2761879210 wait complete. 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 -- start start 2024-04-23T18:32:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.320+0000 7fc6fc82e740 1 Processor -- start 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b659b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41ac020 con 0x557a3b64d490 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41ad490 con 0x557a3b658480 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41ac6f0 con 0x557a3b64ddb0 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6fc82e740 10 client.0 _ll_register_callbacks cb 0x7fc6e4009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.32:44748/0 (socket says 172.21.15.32:44748) 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b659b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2676040978 learned_addr learned my addr 172.21.15.32:0/2676040978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 msgr2=0x557a3b65ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2676040978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 msgr2=0x557a3b659b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b659b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 -- 172.21.15.32:0/2676040978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a3b65f2b0 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6f37fe640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e4005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d967d01bf783eb96 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.321+0000 7fc6d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mon.1 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e401e030 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6e401f040 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mon.1 v2:172.21.15.64:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fc6e4019a40 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mon.1 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc6e4027040 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 10 client.26911 fetch_fsmap learned FSMap version 53 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x557a3b6d3590 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mon.1 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6e401f040 con 0x557a3b64ddb0 2024-04-23T18:32:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mon.1 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fc6e400ec40 con 0x557a3b64ddb0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 10 client.26911 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 10 client.26911 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 20 client.26911 populate_metadata read hostname 'smithi032' 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6d3bd0 0x557a3b6d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 4 client.26911 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "23f38b46", "release": true, "await": true}] 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "set_id": "23f38b46", "release": true, "await": true}) -- 0x557a3b65ad40 con 0x557a3b6d3bd0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.322+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6d3bd0 0x557a3b6d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6d2ffd640 10 client.26911 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6f2ffd640 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6d3bd0 0x557a3b6d5ff0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feb6487467c86695 server_cookie=2b4c73ec236bf5ee in_seq=0 out_seq=0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6d2ffd640 1 -- 172.21.15.32:0/2676040978 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: -1 ) ==== 8+0+590 (crc 0 0 0) 0x557a3b65ad40 con 0x557a3b6d3bd0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6d2ffd640 10 client.26911 handle_command_reply: tid=0 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 client.26911 shutdown 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 msgr2=0x557a3b65a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e4005d30 comp rx=0 tx=0).stop 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 shutdown_connections 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x557a3b6d3bd0 0x557a3b6d5ff0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a3b658480 0x557a3b65ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557a3b64ddb0 0x557a3b65a060 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.323+0000 7fc6fc82e740 1 --2- 172.21.15.32:0/2676040978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557a3b64d490 0x557a3b659b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.324+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 >> 172.21.15.32:0/2676040978 conn(0x557a3b6497b0 msgr2=0x557a3b6489d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.324+0000 7fc6d2ffd640 0 client.26911 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.324+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 shutdown_connections 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.324+0000 7fc6fc82e740 1 -- 172.21.15.32:0/2676040978 wait complete. 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:16.324+0000 7fc6fc82e740 20 client.26911 trim_cache size 0 max 16384 2024-04-23T18:32:16.321 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 53, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 55, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": { 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "23f38b46": { 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 55, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "age_ref": 0.1, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "EXPIRED", 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "timeout": 60.0, 2024-04-23T18:32:16.322 INFO:teuthology.orchestra.run.smithi032.stdout: "expiration": 84.8, 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "members": { 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "file:/": { 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "excluded": false, 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "state": { 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "QUIESCED", 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 84.9 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:16.323 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:32:16.324 INFO:teuthology.orchestra.run.smithi032.stderr:Error EPERM: 2024-04-23T18:32:16.325 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't release set '23f38b46' with rc: 1 (EPERM), stdout: { "epoch": 53, "leader": 24551, "set_version": 55, "sets": { "23f38b46": { "version": 55, "age_ref": 0.1, "state": { "name": "EXPIRED", "age": 0.0 }, "timeout": 60.0, "expiration": 84.8, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCED", "age": 84.9 } } } } } } 2024-04-23T18:32:16.325 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/quiescer.py", line 337, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/quiescer.py", line 313, in do_quiesce raise RuntimeError(f"Error releasing set '{set_id}': {rcinfo}") RuntimeError: Error releasing set '23f38b46': 1 (EPERM) 2024-04-23T18:32:16.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/: 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 data 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.895 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data: 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:total 36 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 . 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 .. 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 base 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 23 18:15 current_logfiles 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 23 18:15 global 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 log 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_commit_ts 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_dynshmem 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 23 18:15 pg_hba.conf 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 23 18:15 pg_ident.conf 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 pg_logical 2024-04-23T18:32:16.896 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 pg_multixact 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_notify 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_replslot 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_serial 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_snapshots 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 pg_stat 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:32 pg_stat_tmp 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:30 pg_subtrans 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_tblspc 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pg_twophase 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 23 18:32 pg_wal 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:24 pg_xact 2024-04-23T18:32:16.897 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 23 18:15 postgresql.auto.conf 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 23 18:15 postgresql.conf 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 23 18:15 postmaster.opts 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base: 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 . 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 1 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 13493 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 23 18:20 13494 2024-04-23T18:32:16.898 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 pgsql_tmp 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:total 7777 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 . 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 .. 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 1247 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:15 1249 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:32:16.899 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:15 1259 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:32:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:32:16.901 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2579 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:32:16.902 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:32:16.903 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:15 2608 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:32:16.904 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2610 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:32:16.905 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 23 18:15 2619 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2619_fsm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:32:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:15 2658 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 2659 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2662 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2663 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2664 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2665 2024-04-23T18:32:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2666 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2667 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:15 2673 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:15 2674 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2678 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2679 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:32:16.908 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2696 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:32:16.909 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2703 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2704 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:32:16.910 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:32:16.911 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:32:16.912 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3455 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:32:16.913 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:32:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:32:16.915 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:32:16.916 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:32:16.917 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:32:16.918 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:32:16.919 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:32:16.919 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:32:16.919 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:32:16.920 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:32:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:total 7777 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 23 18:15 . 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 .. 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 1247 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:32:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:15 1249 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:15 1259 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:32:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:32:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2579 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:32:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:32:16.926 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:15 2608 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:32:16.927 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2610 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:32:16.928 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:32:16.929 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:32:16.930 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 23 18:15 2619 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2619_fsm 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:15 2658 2024-04-23T18:32:16.931 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:15 2659 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2662 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2663 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2664 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2665 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2666 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2667 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:15 2673 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:15 2674 2024-04-23T18:32:16.932 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2678 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2679 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:32:16.933 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2696 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2703 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2704 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:32:16.934 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:32:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:32:16.936 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:32:16.937 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3455 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:32:16.938 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:32:16.939 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:32:16.940 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:32:16.960 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:32:16.960 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:32:16.960 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:32:16.961 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:total 597118 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 23 18:20 . 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 .. 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 112 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 113 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:16 1247 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1247_fsm 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1247_vm 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 23 18:16 1249 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1249_fsm 2024-04-23T18:32:16.962 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1249_vm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 23 18:15 1255 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1255_fsm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1255_vm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 23 18:29 1259 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1259_fsm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1259_vm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 23 18:15 13344 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13344_fsm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13344_vm 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13346 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13348 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349 2024-04-23T18:32:16.963 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13349_fsm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13349_vm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13351 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13353 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13354_fsm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13354_vm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13356 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13358 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 13359_fsm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13359_vm 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 13361 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 13363 2024-04-23T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1417 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 1418 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 443940864 Apr 23 18:30 16396 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 23 18:30 16396_fsm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:19 16396_vm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 2834432 Apr 23 18:30 16397 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:29 16397_fsm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:30 16397_vm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 2146304 Apr 23 18:30 16399 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:29 16399_fsm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:27 16399_vm 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:30 16400 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:29 16402 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 23 18:30 16404 2024-04-23T18:32:16.965 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 82132992 Apr 23 18:30 16406 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:30 16406_fsm 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:28 16406_vm 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 174 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 175 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2187 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2224 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2328 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2336 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2337 2024-04-23T18:32:16.966 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2579 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2600 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2600_fsm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2600_vm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2601_fsm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2601_vm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2602 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2602_fsm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2602_vm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2603 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2603_fsm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2603_vm 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2604 2024-04-23T18:32:16.967 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2605 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2605_fsm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2605_vm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:16 2606 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2606_fsm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2606_vm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2607 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2607_fsm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2607_vm 2024-04-23T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 23 18:16 2608 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2608_fsm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2608_vm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 23 18:15 2609 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2609_fsm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2609_vm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:16 2610 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2610_fsm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2610_vm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2611 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2612_fsm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2612_vm 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2613 2024-04-23T18:32:16.969 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2615_fsm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2615_vm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2616 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2616_fsm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2616_vm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 23 18:15 2617 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2617_fsm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2617_vm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 23 18:15 2618 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2618_fsm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2618_vm 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Apr 23 18:29 2619 2024-04-23T18:32:16.970 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:22 2619_fsm 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2619_vm 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2620 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2650 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2651 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2652 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2653 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2654 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2655 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2656 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2657 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 23 18:16 2658 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 23 18:16 2659 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2660 2024-04-23T18:32:16.971 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2661 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:16 2662 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:16 2663 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2664 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2665 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2666 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2667 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2668 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2669 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2670 2024-04-23T18:32:16.972 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 23 18:16 2673 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 23 18:16 2674 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 23 18:15 2675 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2678 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:16 2679 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2680 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2681 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2682 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2683 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2684 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2685 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2686 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2687 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 2688 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 23 18:15 2689 2024-04-23T18:32:16.973 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 23 18:15 2690 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 23 18:15 2691 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2692 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2693 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:27 2696 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2699 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2701 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2702 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:16 2703 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:16 2704 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2753 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2753_fsm 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2753_vm 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2754 2024-04-23T18:32:16.974 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2755 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2756 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 2757 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2830 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2831 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2832 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2833 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2834 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2835 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2836 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2837 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 23 18:15 2838 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2838_fsm 2024-04-23T18:32:16.975 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2838_vm 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2839 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2840_fsm 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2840_vm 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2841 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2995 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2996 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079 2024-04-23T18:32:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3079_fsm 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3079_vm 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3080 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3081 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:15 3085 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3118 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3119 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 23 18:15 3164 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3256 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3257 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3258 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3350 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3351 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3379 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3380 2024-04-23T18:32:16.977 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3381 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3394 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3394_fsm 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3394_vm 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3395 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3429 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3430 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3431 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3433 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3439 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3440 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 23 18:16 3455 2024-04-23T18:32:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 23 18:15 3456 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3456_fsm 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3456_vm 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3466 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3467 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3468 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3501 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3502 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3503 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3534 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3541_fsm 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3541_vm 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3542 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3574 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3575 2024-04-23T18:32:16.979 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3576 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3596 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3597 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3598 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3599 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3600_fsm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3600_vm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3601_fsm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3601_vm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3602_fsm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3602_vm 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603 2024-04-23T18:32:16.980 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3603_fsm 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3603_vm 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3604 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3605 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3606 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3607 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3608 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 23 18:15 3609 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3712 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 3764_fsm 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3764_vm 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3766 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 3767 2024-04-23T18:32:16.981 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3997 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4143 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4144 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4145 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4146 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4147 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4148 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4149 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4150 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4151 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4152 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4153 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4154 2024-04-23T18:32:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4155 2024-04-23T18:32:16.983 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4156 2024-04-23T18:32:16.983 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4157 2024-04-23T18:32:16.983 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4158 2024-04-23T18:32:16.983 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4159 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4160 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4161 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4162 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4163 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4164 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4165 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4166 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4167 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4168 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4169 2024-04-23T18:32:16.984 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4170 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4171 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4172 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4173 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4174 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 5002 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 548 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 549 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6102 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6104 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6106 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6110 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6111 2024-04-23T18:32:16.985 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6112 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6113 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6117 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 826 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 827 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 828 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 23 18:15 pg_internal.init 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:15 PG_VERSION 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-23T18:32:16.986 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 23 18:15 .. 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/global: 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:total 560 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 23 18:15 . 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1213 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1213_fsm 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1213_vm 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1214 2024-04-23T18:32:16.987 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1214_fsm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1214_vm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 1232 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 1233 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1260 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1260_fsm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1260_vm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1261 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1261_fsm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1261_vm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1262 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 1262_fsm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 1262_vm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2396 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 23 18:15 2396_fsm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2396_vm 2024-04-23T18:32:16.988 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2397 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2671 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2672 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2676 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2677 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2694 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2695 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2697 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 23 18:15 2698 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2846 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2847 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2964 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2965 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 2966 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 2967 2024-04-23T18:32:16.989 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 3592 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 3593 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4060 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4061 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4175 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4176 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4177 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4178 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4181 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4182 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4183 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4184 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 4185 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 4186 2024-04-23T18:32:16.990 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6000 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6001 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6002 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 6100 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6114 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 6115 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:32 pg_control 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 23 18:15 pg_filenode.map 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 23 18:15 pg_internal.init 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/log: 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:total 2 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.991 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1937 Apr 23 18:30 postgresql-Tue.log 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:total 1 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 . 2024-04-23T18:32:16.992 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 mappings 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 23 18:32 replorigin_checkpoint 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 snapshots 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 .. 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 .. 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.993 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 . 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 members 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 offsets 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 .. 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:15 0000 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:total 8 2024-04-23T18:32:16.994 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:15 . 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:15 .. 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 23 18:20 0000 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-23T18:32:16.995 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:total 15 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 23 18:32 . 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 23 18:32 db_0.stat 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 23 18:32 db_13494.stat 2024-04-23T18:32:16.996 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 23 18:32 global.stat 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:32 . 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:total 232 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 23 18:30 . 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 237568 Apr 23 18:30 0017 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.997 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:total 1048576 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 23 18:32 . 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:16.998 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:32 000000010000000000000094 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 000000010000000000000095 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 000000010000000000000096 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 000000010000000000000097 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:27 000000010000000000000098 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:28 000000010000000000000099 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:27 00000001000000000000009A 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:27 00000001000000000000009B 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 00000001000000000000009C 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 00000001000000000000009D 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 00000001000000000000009E 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:28 00000001000000000000009F 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A0 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000A1 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A2 2024-04-23T18:32:16.999 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A3 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000A4 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A5 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000A6 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000A7 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A8 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000A9 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000AA 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000AB 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000AC 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000AD 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000AE 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000AF 2024-04-23T18:32:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000B0 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000B1 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000B2 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000B3 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:25 0000000100000000000000B4 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000B5 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:28 0000000100000000000000B6 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:27 0000000100000000000000B7 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:28 0000000100000000000000B8 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:27 0000000100000000000000B9 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:24 0000000100000000000000BA 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000BB 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000BC 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000BD 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000BE 2024-04-23T18:32:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000BF 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C0 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C1 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C2 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:30 0000000100000000000000C3 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C4 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C5 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:30 0000000100000000000000C6 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C7 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C8 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000C9 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000CA 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000CB 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000CC 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:30 0000000100000000000000CD 2024-04-23T18:32:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000CE 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000CF 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000D0 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000D1 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:29 0000000100000000000000D2 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 23 18:30 0000000100000000000000D3 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 archive_status 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:total 0 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 23 18:15 . 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 23 18:32 .. 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:total 384 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 23 18:24 . 2024-04-23T18:32:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 23 18:32 .. 2024-04-23T18:32:17.004 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 23 18:24 0000 2024-04-23T18:32:17.004 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 23 18:30 0001 2024-04-23T18:32:17.004 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-23T18:32:17.015 INFO:tasks.quiescer:joining Quiescers 2024-04-23T18:32:17.015 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-23T18:32:17.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:32:17.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 -- 172.21.15.32:0/2808182609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102b40 msgr2=0x7f7248106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 --2- 172.21.15.32:0/2808182609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102b40 0x7f7248106b80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f723c009900 tx=0x7f723c02a410 comp rx=0 tx=0).stop 2024-04-23T18:32:17.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 -- 172.21.15.32:0/2808182609 shutdown_connections 2024-04-23T18:32:17.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 --2- 172.21.15.32:0/2808182609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 0x7f724810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 --2- 172.21.15.32:0/2808182609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102b40 0x7f7248106b80 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 --2- 172.21.15.32:0/2808182609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7248102190 0x7f7248102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 -- 172.21.15.32:0/2808182609 >> 172.21.15.32:0/2808182609 conn(0x7f72480fda30 msgr2=0x7f72480ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 -- 172.21.15.32:0/2808182609 shutdown_connections 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.126+0000 7f724ee2a640 1 -- 172.21.15.32:0/2808182609 wait complete. 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 Processor -- start 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 -- start start 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 0x7f7248198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f724819cfc0 0x7f724819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72481121d0 con 0x7f7248102190 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7248112050 con 0x7f724819cfc0 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724ee2a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7248112350 con 0x7f7248107260 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724cbc6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724cbc6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58198/0 (socket says 172.21.15.32:58198) 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f724cbc6640 1 -- 172.21.15.32:0/3793952415 learned_addr learned my addr 172.21.15.32:0/3793952415 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.127+0000 7f7247fff640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 0x7f7248198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724d3c7640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f724819cfc0 0x7f724819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 -- 172.21.15.32:0/3793952415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 msgr2=0x7f7248198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 0x7f7248198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 -- 172.21.15.32:0/3793952415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f724819cfc0 msgr2=0x7f724819d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f724819cfc0 0x7f724819d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 -- 172.21.15.32:0/3793952415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724819dab0 con 0x7f7248102190 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724cbc6640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723800cd80 con 0x7f7248102190 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724819dda0 con 0x7f7248102190 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72380108e0 con 0x7f7248102190 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238018980 con 0x7f7248102190 2024-04-23T18:32:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.128+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72481a5560 con 0x7f7248102190 2024-04-23T18:32:17.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.129+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7248069250 con 0x7f7248102190 2024-04-23T18:32:17.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.130+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7238010430 con 0x7f7248102190 2024-04-23T18:32:17.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.131+0000 7f7245ffb640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 0x7f72300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.131+0000 7f7247fff640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 0x7f72300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.131+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f72380d0c20 con 0x7f7248102190 2024-04-23T18:32:17.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.132+0000 7f7247fff640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 0x7f72300a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f723c006010 tx=0x7f723c000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.136+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f723809d2e0 con 0x7f7248102190 2024-04-23T18:32:17.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.362+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f724810a4f0 con 0x7f7248102190 2024-04-23T18:32:17.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.363+0000 7f7245ffb640 1 -- 172.21.15.32:0/3793952415 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11854 (secure 0 0 0) 0x7f723809cbe0 con 0x7f7248102190 2024-04-23T18:32:17.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:17.356 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:30:23.814504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24515,24551,24499]},"id":1}]} 2024-04-23T18:32:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 53 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 msgr2=0x7f72300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 0x7f72300a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f723c006010 tx=0x7f723c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 msgr2=0x7f7248198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 shutdown_connections 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f724819cfc0 0x7f724819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7248107260 0x7f7248198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f72300a3d20 0x7f72300a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 --2- 172.21.15.32:0/3793952415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7248102190 0x7f7248198750 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 >> 172.21.15.32:0/3793952415 conn(0x7f72480fda30 msgr2=0x7f72480fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 shutdown_connections 2024-04-23T18:32:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.365+0000 7f724ee2a640 1 -- 172.21.15.32:0/3793952415 wait complete. 2024-04-23T18:32:17.369 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24551 quiesce db --cancel --all' 2024-04-23T18:32:17.370 DEBUG:teuthology.orchestra.run.smithi032:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24551 quiesce db --cancel --all 2024-04-23T18:32:17.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 -- 172.21.15.32:0/820580685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 msgr2=0x7f182c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 --2- 172.21.15.32:0/820580685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c102650 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f181c009930 tx=0x7f181c02a420 comp rx=0 tx=0).stop 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 -- 172.21.15.32:0/820580685 shutdown_connections 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 --2- 172.21.15.32:0/820580685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c1044f0 0x7f182c10eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 --2- 172.21.15.32:0/820580685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c103b50 0x7f182c103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 --2- 172.21.15.32:0/820580685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c102650 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 -- 172.21.15.32:0/820580685 >> 172.21.15.32:0/820580685 conn(0x7f182c0fda10 msgr2=0x7f182c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 -- 172.21.15.32:0/820580685 shutdown_connections 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 -- 172.21.15.32:0/820580685 wait complete. 2024-04-23T18:32:17.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.484+0000 7f18320c2640 1 Processor -- start 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 -- start start 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 0x7f182c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 0x7f182c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f182c112270 con 0x7f182c102270 2024-04-23T18:32:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1120f0 con 0x7f182c1044f0 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f18320c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1123f0 con 0x7f182c103b50 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58222/0 (socket says 172.21.15.32:58222) 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 -- 172.21.15.32:0/2835522049 learned_addr learned my addr 172.21.15.32:0/2835522049 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182bfff640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 0x7f182c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182affd640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 0x7f182c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 -- 172.21.15.32:0/2835522049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 msgr2=0x7f182c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 0x7f182c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 -- 172.21.15.32:0/2835522049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 msgr2=0x7f182c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 0x7f182c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182b7fe640 1 -- 172.21.15.32:0/2835522049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182c198d70 con 0x7f182c102270 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182affd640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 0x7f182c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.485+0000 7f182bfff640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 0x7f182c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f181c004770 tx=0x7f181c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f1828ff9640 1 -- 172.21.15.32:0/2835522049 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c03f070 con 0x7f182c102270 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182c199000 con 0x7f182c102270 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f1828ff9640 1 -- 172.21.15.32:0/2835522049 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c03b460 con 0x7f182c102270 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f1828ff9640 1 -- 172.21.15.32:0/2835522049 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c035070 con 0x7f182c102270 2024-04-23T18:32:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.486+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182c1a5300 con 0x7f182c102270 2024-04-23T18:32:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.488+0000 7f1828ff9640 1 -- 172.21.15.32:0/2835522049 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f181c03b8d0 con 0x7f182c102270 2024-04-23T18:32:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.488+0000 7f1834dd4740 1 Processor -- start 2024-04-23T18:32:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 -- start start 2024-04-23T18:32:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560d521d130 con 0x5560d52b7360 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5560d51b9c50 con 0x5560d52c2380 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1834dd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5560d52190f0 con 0x5560d52b7d10 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58224/0 (socket says 172.21.15.32:58224) 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182bfff640 1 -- 172.21.15.32:0/106017931 learned_addr learned my addr 172.21.15.32:0/106017931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1828ff9640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 0x7f18140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182b7fe640 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182bfff640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 0x7f18140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 -- 172.21.15.32:0/106017931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 msgr2=0x5560d52c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52c1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 -- 172.21.15.32:0/106017931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 msgr2=0x5560d52b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f1828ff9640 1 -- 172.21.15.32:0/2835522049 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f181c0f0850 con 0x7f182c102270 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.489+0000 7f182affd640 1 -- 172.21.15.32:0/106017931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560d51d3fd0 con 0x5560d52c2380 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f182affd640 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c47d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f182c100170 tx=0x7f182000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1f5a1e962deb1da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f182bfff640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 0x7f18140a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f182400afd0 tx=0x7f1824004ab0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1811ffb640 1 -- 172.21.15.32:0/106017931 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820015070 con 0x5560d52c2380 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1811ffb640 1 -- 172.21.15.32:0/106017931 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f182001a910 con 0x5560d52c2380 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1811ffb640 1 -- 172.21.15.32:0/106017931 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820011980 con 0x5560d52c2380 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 -- 172.21.15.32:0/106017931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 msgr2=0x5560d52c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c47d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f182c100170 tx=0x7f182000f730 comp rx=0 tx=0).stop 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 -- 172.21.15.32:0/106017931 shutdown_connections 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c47d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 --2- 172.21.15.32:0/106017931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 -- 172.21.15.32:0/106017931 >> 172.21.15.32:0/106017931 conn(0x5560d52b3620 msgr2=0x5560d52b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 -- 172.21.15.32:0/106017931 shutdown_connections 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.490+0000 7f1834dd4740 1 -- 172.21.15.32:0/106017931 wait complete. 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 -- start start 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 Processor -- start 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7360 0x5560d52bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560d52c52f0 con 0x5560d52c2380 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5560d51b7640 con 0x5560d52b7d10 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5560d52c5470 con 0x5560d52b7360 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f1834dd4740 10 client.0 _ll_register_callbacks cb 0x5560d51b7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58234/0 (socket says 172.21.15.32:58234) 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f182bfff640 1 -- 172.21.15.32:0/2863092404 learned_addr learned my addr 172.21.15.32:0/2863092404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.491+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7360 0x5560d52bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182affd640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 -- 172.21.15.32:0/2863092404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7360 msgr2=0x5560d52bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7360 0x5560d52bf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 -- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 msgr2=0x5560d52bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 -- 172.21.15.32:0/2863092404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560d4ffba20 con 0x5560d52c2380 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f182bfff640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1824013fe0 tx=0x7f1824004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe1a71d2fc0d692 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f180affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182401d800 con 0x5560d52c2380 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f182401d9a0 con 0x5560d52c2380 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f182001a910 con 0x5560d52c2380 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f182401dba0 con 0x5560d52c2380 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f1834dd4740 10 client.17445 fetch_fsmap learned FSMap version 53 2024-04-23T18:32:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5560d505c1b0 con 0x5560d52c2380 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.492+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f182001a910 con 0x5560d52c2380 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f182402e070 con 0x5560d52c2380 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 10 client.17445 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 10 client.17445 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 20 client.17445 populate_metadata read hostname 'smithi032' 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d52d2990 0x5560d52d4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 4 client.17445 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560d522ab50 con 0x5560d52d2990 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d52d2990 0x5560d52d4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f182b7fe640 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d52d2990 0x5560d52d4e30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7adf40884290473d server_cookie=70d3f899957e24d9 in_seq=0 out_seq=0 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.493+0000 7f180affd640 10 client.17445 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f180affd640 1 -- 172.21.15.32:0/2863092404 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560d522ab50 con 0x5560d52d2990 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f180affd640 10 client.17445 handle_command_reply: tid=0 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 client.17445 shutdown 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 msgr2=0x5560d52bfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1824013fe0 tx=0x7f1824004a40 comp rx=0 tx=0).stop 2024-04-23T18:32:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 shutdown_connections 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d52d2990 0x5560d52d4e30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c2380 0x5560d52bfd70 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f180affd640 0 client.17445 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 --2- 172.21.15.32:0/2863092404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7360 0x5560d52bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 >> 172.21.15.32:0/2863092404 conn(0x5560d52b3620 msgr2=0x5560d52b9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 shutdown_connections 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.494+0000 7f1834dd4740 1 -- 172.21.15.32:0/2863092404 wait complete. 2024-04-23T18:32:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.495+0000 7f1834dd4740 20 client.17445 trim_cache size 0 max 16384 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 Processor -- start 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 -- start start 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52c2380 0x5560d52da700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1b03f0 con 0x5560d52b7360 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1adc60 con 0x5560d52b7d10 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f1834dd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1abad0 con 0x5560d52c2380 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58238/0 (socket says 172.21.15.32:58238) 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 -- 172.21.15.32:0/1809295823 learned_addr learned my addr 172.21.15.32:0/1809295823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 -- 172.21.15.32:0/1809295823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52c2380 msgr2=0x5560d52da700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52c2380 0x5560d52da700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182affd640 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 -- 172.21.15.32:0/1809295823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 msgr2=0x5560d52bf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 -- 172.21.15.32:0/1809295823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560d4eef310 con 0x5560d52b7360 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182affd640 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.512+0000 7f182b7fe640 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ee5c0 tx=0x7f181c08e250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0d95f5c5adac4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f18117fa640 1 -- 172.21.15.32:0/1809295823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c048040 con 0x5560d52b7360 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f18117fa640 1 -- 172.21.15.32:0/1809295823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c0bcf10 con 0x5560d52b7360 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f18117fa640 1 -- 172.21.15.32:0/1809295823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c03b610 con 0x5560d52b7360 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 -- 172.21.15.32:0/1809295823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 msgr2=0x5560d52ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ee5c0 tx=0x7f181c08e250 comp rx=0 tx=0).stop 2024-04-23T18:32:17.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:17 smithi032 ceph-mon[29221]: pgmap v541: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2024-04-23T18:32:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 -- 172.21.15.32:0/1809295823 shutdown_connections 2024-04-23T18:32:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52c2380 0x5560d52da700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52b7d10 0x5560d52bf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 --2- 172.21.15.32:0/1809295823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52b7360 0x5560d52ce1b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 -- 172.21.15.32:0/1809295823 >> 172.21.15.32:0/1809295823 conn(0x5560d52b3620 msgr2=0x5560d52bf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 -- 172.21.15.32:0/1809295823 shutdown_connections 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.513+0000 7f1834dd4740 1 -- 172.21.15.32:0/1809295823 wait complete. 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 -- start start 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 Processor -- start 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1ac120 con 0x5560d52c7090 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1ad590 con 0x5560d52c2380 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1ac7f0 con 0x5560d52b7d10 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f1834dd4740 10 client.0 _ll_register_callbacks cb 0x7f1824011f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f182bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58252/0 (socket says 172.21.15.32:58252) 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.514+0000 7f182bfff640 1 -- 172.21.15.32:0/3050817264 learned_addr learned my addr 172.21.15.32:0/3050817264 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182affd640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182b7fe640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 -- 172.21.15.32:0/3050817264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 msgr2=0x5560d52ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 -- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 msgr2=0x5560d52c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 -- 172.21.15.32:0/3050817264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560d5338f80 con 0x5560d52c7090 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f182bfff640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f18240118b0 tx=0x7f182400a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421fc2165d80a6c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f180affd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824034070 con 0x5560d52c7090 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f1824004b20 con 0x5560d52c7090 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181c0bcf10 con 0x5560d52c7090 2024-04-23T18:32:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1824004d60 con 0x5560d52c7090 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f1834dd4740 10 client.17451 fetch_fsmap learned FSMap version 53 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5560d533a090 con 0x5560d52c7090 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.515+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c0bcf10 con 0x5560d52c7090 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f182402e070 con 0x5560d52c7090 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 10 client.17451 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 10 client.17451 resolve_mds: validated gid 24551 aka daemon mds.b 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 20 client.17451 populate_metadata read hostname 'smithi032' 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d533a730 0x5560d533cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 4 client.17451 mds_command: new command op to 24551 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 --> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5560d52249c0 con 0x5560d533a730 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f182b7fe640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d533a730 0x5560d533cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f182b7fe640 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d533a730 0x5560d533cb50 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88dbe35f14ebae63 server_cookie=9b7b6e161b05c109 in_seq=0 out_seq=0 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f180affd640 10 client.17451 ms_handle_connect on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f180affd640 1 -- 172.21.15.32:0/3050817264 <== mds.0 v2:172.21.15.40:6838/2479383965 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x5560d52249c0 con 0x5560d533a730 2024-04-23T18:32:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.516+0000 7f180affd640 10 client.17451 handle_command_reply: tid=0 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 client.17451 shutdown 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 msgr2=0x5560d52c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f18240118b0 tx=0x7f182400a550 comp rx=0 tx=0).stop 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 shutdown_connections 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:6838/2479383965,v1:172.21.15.40:6839/2479383965] conn(0x5560d533a730 0x5560d533cb50 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560d52c7090 0x5560d52c7510 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560d52c2380 0x5560d52c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f180affd640 0 client.17451 ms_handle_reset on v2:172.21.15.40:6838/2479383965 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 --2- 172.21.15.32:0/3050817264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560d52b7d10 0x5560d52ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 >> 172.21.15.32:0/3050817264 conn(0x5560d52b3620 msgr2=0x5560d52bfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 shutdown_connections 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 1 -- 172.21.15.32:0/3050817264 wait complete. 2024-04-23T18:32:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f1834dd4740 20 client.17451 trim_cache size 0 max 16384 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 53, 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "leader": 24551, 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "set_version": 55, 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "sets": {} 2024-04-23T18:32:17.516 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 msgr2=0x7f18140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 0x7f18140a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f182400afd0 tx=0x7f1824004ab0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 msgr2=0x7f182c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f181c004770 tx=0x7f181c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 shutdown_connections 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f182c1044f0 0x7f182c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f182c103b50 0x7f182c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f18140a3d20 0x7f18140a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 --2- 172.21.15.32:0/2835522049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f182c102270 0x7f182c19e440 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.517+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 >> 172.21.15.32:0/2835522049 conn(0x7f182c0fda10 msgr2=0x7f182c0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.518+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 shutdown_connections 2024-04-23T18:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.518+0000 7f18320c2640 1 -- 172.21.15.32:0/2835522049 wait complete. 2024-04-23T18:32:17.521 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/quiescer.py", line 408, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/quiescer.py", line 361, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error releasing set '23f38b46': 1 (EPERM) 2024-04-23T18:32:17.521 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-23T18:32:17.531 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:32:17.539 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-23T18:32:17.648 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-23T18:32:17.648 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:32:17.648 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:32:17.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-23T18:32:17.649 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:17 smithi040 ceph-mon[32626]: pgmap v541: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2024-04-23T18:32:17.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:17 smithi064 ceph-mon[33062]: pgmap v541: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2024-04-23T18:32:17.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-23T18:32:17.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/2567102970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 msgr2=0x7fd2a0102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/2567102970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0102590 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd290009900 tx=0x7fd29002a410 comp rx=0 tx=0).stop 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/2567102970 shutdown_connections 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/2567102970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 0x7fd2a010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/2567102970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a0102b60 0x7fd2a0106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/2567102970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0102590 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/2567102970 >> 172.21.15.32:0/2567102970 conn(0x7fd2a00fda10 msgr2=0x7fd2a00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/2567102970 shutdown_connections 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/2567102970 wait complete. 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.893+0000 7fd2a5ab4640 1 Processor -- start 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 -- start start 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 0x7fd2a0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 0x7fd2a019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a01121e0 con 0x7fd2a01021b0 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0112060 con 0x7fd2a0107280 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd2a5ab4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0112360 con 0x7fd2a019d000 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58260/0 (socket says 172.21.15.32:58260) 2024-04-23T18:32:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 -- 172.21.15.32:0/1047672990 learned_addr learned my addr 172.21.15.32:0/1047672990 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29effd640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 0x7fd2a0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29ffff640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 0x7fd2a019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 -- 172.21.15.32:0/1047672990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 msgr2=0x7fd2a019d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 0x7fd2a019d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 -- 172.21.15.32:0/1047672990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 msgr2=0x7fd2a0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 0x7fd2a0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 -- 172.21.15.32:0/1047672990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a019daf0 con 0x7fd2a01021b0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29ffff640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 0x7fd2a019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.894+0000 7fd29f7fe640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd290005940 tx=0x7fd2900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.895+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29003f030 con 0x7fd2a01021b0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.895+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd290003cb0 con 0x7fd2a01021b0 2024-04-23T18:32:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.895+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a019dd80 con 0x7fd2a01021b0 2024-04-23T18:32:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.895+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290035070 con 0x7fd2a01021b0 2024-04-23T18:32:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.895+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a01a5540 con 0x7fd2a01021b0 2024-04-23T18:32:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.896+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a0050530 con 0x7fd2a01021b0 2024-04-23T18:32:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.897+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd290042030 con 0x7fd2a01021b0 2024-04-23T18:32:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.897+0000 7fd29cff9640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 0x7fd2880a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.898+0000 7fd29effd640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 0x7fd2880a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.898+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fd2900f12d0 con 0x7fd2a01021b0 2024-04-23T18:32:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.898+0000 7fd29effd640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 0x7fd2880a61d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0101170 tx=0x7fd29400b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:17.901+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2900bd990 con 0x7fd2a01021b0 2024-04-23T18:32:18.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.110+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd2a0102b60 con 0x7fd2880a3d10 2024-04-23T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.119+0000 7fd29cff9640 1 -- 172.21.15.32:0/1047672990 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fd2a0102b60 con 0x7fd2880a3d10 2024-04-23T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:15:08", "first": "2024-04-23T18:16:00", "last": "2024-04-23T18:32:00", "last_pruned": "2024-04-23T18:32:00", "created_count": 17, "pruned_count": 11, "active": true} 2024-04-23T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 msgr2=0x7fd2880a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 0x7fd2880a61d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0101170 tx=0x7fd29400b390 comp rx=0 tx=0).stop 2024-04-23T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 msgr2=0x7fd2a0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd290005940 tx=0x7fd2900042f0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 shutdown_connections 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2a019d000 0x7fd2a019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2a0107280 0x7fd2a0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fd2880a3d10 0x7fd2880a61d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 --2- 172.21.15.32:0/1047672990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2a01021b0 0x7fd2a0198740 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 >> 172.21.15.32:0/1047672990 conn(0x7fd2a00fda10 msgr2=0x7fd2a00fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 shutdown_connections 2024-04-23T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.121+0000 7fd2a5ab4640 1 -- 172.21.15.32:0/1047672990 wait complete. 2024-04-23T18:32:18.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2514898855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec0106190 msgr2=0x7f0ec010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2514898855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec0106190 0x7f0ec010dd60 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009ca0 tx=0x7f0eb802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2514898855 shutdown_connections 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2514898855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec0106190 0x7f0ec010dd60 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2514898855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0101a70 0x7f0ec0105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2514898855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec01010c0 0x7f0ec01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2514898855 >> 172.21.15.32:0/2514898855 conn(0x7f0ec0077fd0 msgr2=0x7f0ec00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2514898855 shutdown_connections 2024-04-23T18:32:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.237+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2514898855 wait complete. 2024-04-23T18:32:18.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 Processor -- start 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 -- start start 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0106190 0x7f0ec0196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 0x7f0ec019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec01100c0 con 0x7f0ec01010c0 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec010ff40 con 0x7f0ec019af00 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ec51a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0110240 con 0x7f0ec0106190 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58292/0 (socket says 172.21.15.32:58292) 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 -- 172.21.15.32:0/2761911622 learned_addr learned my addr 172.21.15.32:0/2761911622 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebf59e640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 0x7f0ec019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0eb7fff640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0106190 0x7f0ec0196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 -- 172.21.15.32:0/2761911622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0106190 msgr2=0x7f0ec0196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0106190 0x7f0ec0196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 -- 172.21.15.32:0/2761911622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 msgr2=0x7f0ec019b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 0x7f0ec019b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.238+0000 7f0ebed9d640 1 -- 172.21.15.32:0/2761911622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec019b9f0 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ebf59e640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 0x7f0ec019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ebed9d640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000b4a0 tx=0x7f0eb000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0014070 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec019bce0 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eb0010ac0 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0017c30 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.239+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec01a34d0 con 0x7f0ec01010c0 2024-04-23T18:32:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.240+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec0102590 con 0x7f0ec01010c0 2024-04-23T18:32:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.241+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0eb0017070 con 0x7f0ec01010c0 2024-04-23T18:32:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.242+0000 7f0ebcd99640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 0x7f0eac0a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.242+0000 7f0eb7fff640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 0x7f0eac0a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.242+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f0eb00cfeb0 con 0x7f0ec01010c0 2024-04-23T18:32:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.242+0000 7f0eb7fff640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 0x7f0eac0a83e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0100a20 tx=0x7f0ea8008040 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.245+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0eb00feff0 con 0x7f0ec01010c0 2024-04-23T18:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.455+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 --> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f0ec0101a70 con 0x7f0eac0a5f20 2024-04-23T18:32:18.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.464+0000 7f0ebcd99640 1 -- 172.21.15.32:0/2761911622 <== mgr.15294 v2:172.21.15.32:6800/3410706145 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f0ec0101a70 con 0x7f0eac0a5f20 2024-04-23T18:32:18.456 INFO:teuthology.orchestra.run.smithi032.stdout:/ 1m 6m3h 2024-04-23T18:32:18.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 msgr2=0x7f0eac0a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 0x7f0eac0a83e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0100a20 tx=0x7f0ea8008040 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 msgr2=0x7f0ec0196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000b4a0 tx=0x7f0eb000b960 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 shutdown_connections 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ec019af00 0x7f0ec019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ec0106190 0x7f0ec0196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f0eac0a5f20 0x7f0eac0a83e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 --2- 172.21.15.32:0/2761911622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec01010c0 0x7f0ec0196640 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 >> 172.21.15.32:0/2761911622 conn(0x7f0ec0077fd0 msgr2=0x7f0ec00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 shutdown_connections 2024-04-23T18:32:18.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.466+0000 7f0ec51a2640 1 -- 172.21.15.32:0/2761911622 wait complete. 2024-04-23T18:32:18.468 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:18 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3793952415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:18.470 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-23T18:32:18.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 -- 172.21.15.32:0/1871926174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18104d50 msgr2=0x7f4c18108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/1871926174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18104d50 0x7f4c18108d90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009900 tx=0x7f4c0c02a410 comp rx=0 tx=0).stop 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 -- 172.21.15.32:0/1871926174 shutdown_connections 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/1871926174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18109470 0x7f4c18111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/1871926174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18104d50 0x7f4c18108d90 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/1871926174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 0x7f4c18104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.594+0000 7f4c1f38e640 1 -- 172.21.15.32:0/1871926174 >> 172.21.15.32:0/1871926174 conn(0x7f4c180ffc00 msgr2=0x7f4c18102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- 172.21.15.32:0/1871926174 shutdown_connections 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- 172.21.15.32:0/1871926174 wait complete. 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 Processor -- start 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- start start 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 0x7f4c181a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 0x7f4c1819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4c181143b0 con 0x7f4c18109470 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c18114230 con 0x7f4c1819a7f0 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1f38e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4c18114530 con 0x7f4c181043a0 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1c929640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1c929640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58316/0 (socket says 172.21.15.32:58316) 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.595+0000 7f4c1c929640 1 -- 172.21.15.32:0/3261871524 learned_addr learned my addr 172.21.15.32:0/3261871524 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1d12a640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 0x7f4c181a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1d92b640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 0x7f4c1819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 -- 172.21.15.32:0/3261871524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 msgr2=0x7f4c181a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 0x7f4c181a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 -- 172.21.15.32:0/3261871524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 msgr2=0x7f4c1819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 0x7f4c1819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 -- 172.21.15.32:0/3261871524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1819b380 con 0x7f4c18109470 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1d92b640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 0x7f4c1819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1c929640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c03f030 con 0x7f4c18109470 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1819b610 con 0x7f4c18109470 2024-04-23T18:32:18.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c0c003aa0 con 0x7f4c18109470 2024-04-23T18:32:18.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c035070 con 0x7f4c18109470 2024-04-23T18:32:18.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.596+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c181a7750 con 0x7f4c18109470 2024-04-23T18:32:18.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.597+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c18105870 con 0x7f4c18109470 2024-04-23T18:32:18.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.598+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4c0c042030 con 0x7f4c18109470 2024-04-23T18:32:18.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.599+0000 7f4c167fc640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 0x7f4c000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.599+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f4c0c0f1040 con 0x7f4c18109470 2024-04-23T18:32:18.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.599+0000 7f4c1d12a640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 0x7f4c000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.600+0000 7f4c1d12a640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 0x7f4c000a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009bd0 tx=0x7f4c10005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.603+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c0c0bd700 con 0x7f4c18109470 2024-04-23T18:32:18.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:18 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3793952415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:18.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:18 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3793952415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:18.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.791+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c1810c6a0 con 0x7f4c18109470 2024-04-23T18:32:18.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.793+0000 7f4c167fc640 1 -- 172.21.15.32:0/3261871524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23004 (secure 0 0 0) 0x7f4c0c09be20 con 0x7f4c18109470 2024-04-23T18:32:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:18.786 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:32:10.775144+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:32:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 msgr2=0x7f4c000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 0x7f4c000a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009bd0 tx=0x7f4c10005f90 comp rx=0 tx=0).stop 2024-04-23T18:32:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 msgr2=0x7f4c181a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c004060 comp rx=0 tx=0).stop 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 shutdown_connections 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c1819a7f0 0x7f4c1819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f4c000a3d20 0x7f4c000a61e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c18109470 0x7f4c181a0b70 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 --2- 172.21.15.32:0/3261871524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c181043a0 0x7f4c181a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 >> 172.21.15.32:0/3261871524 conn(0x7f4c180ffc00 msgr2=0x7f4c18100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 shutdown_connections 2024-04-23T18:32:18.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.795+0000 7f4c1f38e640 1 -- 172.21.15.32:0/3261871524 wait complete. 2024-04-23T18:32:18.799 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:32:18.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 -- 172.21.15.32:0/1199050162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606810c6d0 msgr2=0x7f606810eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 --2- 172.21.15.32:0/1199050162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606810c6d0 0x7f606810eb60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6060009ca0 tx=0x7f606002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 -- 172.21.15.32:0/1199050162 shutdown_connections 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 --2- 172.21.15.32:0/1199050162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606810c6d0 0x7f606810eb60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 --2- 172.21.15.32:0/1199050162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6068103b70 0x7f606810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 --2- 172.21.15.32:0/1199050162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60681031c0 0x7f60681035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 -- 172.21.15.32:0/1199050162 >> 172.21.15.32:0/1199050162 conn(0x7f6068077f70 msgr2=0x7f6068078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 -- 172.21.15.32:0/1199050162 shutdown_connections 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.912+0000 7f606e15d640 1 -- 172.21.15.32:0/1199050162 wait complete. 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 Processor -- start 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 -- start start 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681031c0 0x7f606819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f606810c6d0 0x7f606819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6068112380 con 0x7f60681986e0 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6068112200 con 0x7f60681031c0 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f606e15d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6068112500 con 0x7f606810c6d0 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f6067fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f6067fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58338/0 (socket says 172.21.15.32:58338) 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.913+0000 7f6067fff640 1 -- 172.21.15.32:0/971489085 learned_addr learned my addr 172.21.15.32:0/971489085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f60677fe640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681031c0 0x7f606819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6066ffd640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f606810c6d0 0x7f606819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 -- 172.21.15.32:0/971489085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f606810c6d0 msgr2=0x7f606819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f606810c6d0 0x7f606819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 -- 172.21.15.32:0/971489085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681031c0 msgr2=0x7f606819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681031c0 0x7f606819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 -- 172.21.15.32:0/971489085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60681992e0 con 0x7f60681986e0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6067fff640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6060005cc0 tx=0x7f606002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060003bf0 con 0x7f60681986e0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6060003d90 con 0x7f60681986e0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6068199570 con 0x7f60681986e0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606002fbc0 con 0x7f60681986e0 2024-04-23T18:32:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.914+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60681a5710 con 0x7f60681986e0 2024-04-23T18:32:18.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.915+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606806ae40 con 0x7f60681986e0 2024-04-23T18:32:18.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.916+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f606002fd60 con 0x7f60681986e0 2024-04-23T18:32:18.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.917+0000 7f6064ff9640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 0x7f60500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.917+0000 7f60677fe640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 0x7f60500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.918+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f60600f1f90 con 0x7f60681986e0 2024-04-23T18:32:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.918+0000 7f60677fe640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 0x7f60500a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f605800bbe0 tx=0x7f605800b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:18.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:18.921+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60600be760 con 0x7f60681986e0 2024-04-23T18:32:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.107+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6068103b70 con 0x7f60681986e0 2024-04-23T18:32:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.107+0000 7f6064ff9640 1 -- 172.21.15.32:0/971489085 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f60600be060 con 0x7f60681986e0 2024-04-23T18:32:19.100 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:32:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 msgr2=0x7f60500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 0x7f60500a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f605800bbe0 tx=0x7f605800b390 comp rx=0 tx=0).stop 2024-04-23T18:32:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 msgr2=0x7f6068198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6060005cc0 tx=0x7f606002f380 comp rx=0 tx=0).stop 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 shutdown_connections 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f60500a3d20 0x7f60500a61e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60681986e0 0x7f6068198b60 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f606810c6d0 0x7f606819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 --2- 172.21.15.32:0/971489085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681031c0 0x7f606819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 >> 172.21.15.32:0/971489085 conn(0x7f6068077f70 msgr2=0x7f60680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 shutdown_connections 2024-04-23T18:32:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.110+0000 7f606e15d640 1 -- 172.21.15.32:0/971489085 wait complete. 2024-04-23T18:32:19.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:32:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- 172.21.15.32:0/2086415289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 msgr2=0x7f0928101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- 172.21.15.32:0/2086415289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f0928101430 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0918009900 tx=0x7f091802a410 comp rx=0 tx=0).stop 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- 172.21.15.32:0/2086415289 shutdown_connections 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- 172.21.15.32:0/2086415289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0928106120 0x7f092810dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- 172.21.15.32:0/2086415289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0928101a00 0x7f0928105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- 172.21.15.32:0/2086415289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f0928101430 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- 172.21.15.32:0/2086415289 >> 172.21.15.32:0/2086415289 conn(0x7f0928077fa0 msgr2=0x7f09280783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- 172.21.15.32:0/2086415289 shutdown_connections 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- 172.21.15.32:0/2086415289 wait complete. 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 Processor -- start 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- start start 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0928106120 0x7f092819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09281985f0 0x7f0928198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f09281110e0 con 0x7f0928101050 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0928110f60 con 0x7f0928106120 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.228+0000 7f092f115640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0928111260 con 0x7f09281985f0 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58356/0 (socket says 172.21.15.32:58356) 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 -- 172.21.15.32:0/785613559 learned_addr learned my addr 172.21.15.32:0/785613559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092d6b2640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09281985f0 0x7f0928198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 -- 172.21.15.32:0/785613559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09281985f0 msgr2=0x7f0928198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09281985f0 0x7f0928198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 -- 172.21.15.32:0/785613559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0928106120 msgr2=0x7f092819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0928106120 0x7f092819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 -- 172.21.15.32:0/785613559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0928199110 con 0x7f0928101050 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092ceb1640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f0918005ab0 tx=0x7f0918003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091803f070 con 0x7f0928101050 2024-04-23T18:32:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0918031d60 con 0x7f0928101050 2024-04-23T18:32:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09281993a0 con 0x7f0928101050 2024-04-23T18:32:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0918035030 con 0x7f0928101050 2024-04-23T18:32:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.229+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a5590 con 0x7f0928101050 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.231+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0928102520 con 0x7f0928101050 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.232+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0918042070 con 0x7f0928101050 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.232+0000 7f0925ffb640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 0x7f09100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.233+0000 7f0927fff640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 0x7f09100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.233+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f09180f10b0 con 0x7f0928101050 2024-04-23T18:32:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.233+0000 7f0927fff640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 0x7f09100a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f091c005fd0 tx=0x7f091c005850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.237+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09180bd810 con 0x7f0928101050 2024-04-23T18:32:19.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.422+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0928199b20 con 0x7f0928101050 2024-04-23T18:32:19.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.423+0000 7f0925ffb640 1 -- 172.21.15.32:0/785613559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f09180bd110 con 0x7f0928101050 2024-04-23T18:32:19.416 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 msgr2=0x7f09100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 0x7f09100a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f091c005fd0 tx=0x7f091c005850 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 msgr2=0x7f092819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f0918005ab0 tx=0x7f0918003cd0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 shutdown_connections 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09281985f0 0x7f0928198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0928106120 0x7f092819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f09100a3d20 0x7f09100a61e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 --2- 172.21.15.32:0/785613559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0928101050 0x7f092819e440 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 >> 172.21.15.32:0/785613559 conn(0x7f0928077fa0 msgr2=0x7f0928069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 shutdown_connections 2024-04-23T18:32:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.425+0000 7f092f115640 1 -- 172.21.15.32:0/785613559 wait complete. 2024-04-23T18:32:19.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:19 smithi032 ceph-mon[29221]: pgmap v542: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-23T18:32:19.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:19 smithi032 ceph-mon[29221]: from='client.17457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:19.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3261871524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:19.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:19 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/971489085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:19.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 -- 172.21.15.32:0/1781567370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8101180 msgr2=0x7f1bb8101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/1781567370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8101180 0x7f1bb8101560 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009930 tx=0x7f1bb402a420 comp rx=0 tx=0).stop 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 -- 172.21.15.32:0/1781567370 shutdown_connections 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/1781567370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8103400 0x7f1bb810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/1781567370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8102a60 0x7f1bb8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/1781567370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8101180 0x7f1bb8101560 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 -- 172.21.15.32:0/1781567370 >> 172.21.15.32:0/1781567370 conn(0x7f1bb8077f70 msgr2=0x7f1bb8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 -- 172.21.15.32:0/1781567370 shutdown_connections 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.541+0000 7f1bc04bb640 1 -- 172.21.15.32:0/1781567370 wait complete. 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 Processor -- start 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 -- start start 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 0x7f1bb81a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 0x7f1bb81a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81111a0 con 0x7f1bb8103400 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8111020 con 0x7f1bb8102a60 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bc04bb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8111320 con 0x7f1bb8101180 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bbea58640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bbea58640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58378/0 (socket says 172.21.15.32:58378) 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.542+0000 7f1bbea58640 1 -- 172.21.15.32:0/2811806495 learned_addr learned my addr 172.21.15.32:0/2811806495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbe257640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 0x7f1bb81a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbda56640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 0x7f1bb81a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 -- 172.21.15.32:0/2811806495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 msgr2=0x7f1bb81a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 0x7f1bb81a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 -- 172.21.15.32:0/2811806495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 msgr2=0x7f1bb81a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 0x7f1bb81a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 -- 172.21.15.32:0/2811806495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb819b000 con 0x7f1bb8103400 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbe257640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 0x7f1bb81a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbda56640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 0x7f1bb81a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bbea58640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cad0 tx=0x7f1bb000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000ae20 con 0x7f1bb8103400 2024-04-23T18:32:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bb0005020 con 0x7f1bb8103400 2024-04-23T18:32:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000f6b0 con 0x7f1bb8103400 2024-04-23T18:32:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb819b2f0 con 0x7f1bb8103400 2024-04-23T18:32:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.543+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb81a7510 con 0x7f1bb8103400 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.544+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb8102570 con 0x7f1bb8103400 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.545+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1bb00051d0 con 0x7f1bb8103400 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.546+0000 7f1baf7fe640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 0x7f1ba40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.546+0000 7f1bbe257640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 0x7f1ba40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.546+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f1bb00d09c0 con 0x7f1bb8103400 2024-04-23T18:32:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.547+0000 7f1bbe257640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 0x7f1ba40a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000c00 tx=0x7f1bb4031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.551+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bb0014030 con 0x7f1bb8103400 2024-04-23T18:32:19.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:19 smithi040 ceph-mon[32626]: pgmap v542: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-23T18:32:19.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:19 smithi040 ceph-mon[32626]: from='client.17457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:19.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:19 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3261871524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:19.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:19 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/971489085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:19.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:19 smithi064 ceph-mon[33062]: pgmap v542: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-23T18:32:19.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:19 smithi064 ceph-mon[33062]: from='client.17457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:19.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:19 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3261871524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:19.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:19 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/971489085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.739+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1bb81a7740 con 0x7f1bb8103400 2024-04-23T18:32:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.739+0000 7f1baf7fe640 1 -- 172.21.15.32:0/2811806495 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f1bb009d020 con 0x7f1bb8103400 2024-04-23T18:32:19.732 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 msgr2=0x7f1ba40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 0x7f1ba40a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000c00 tx=0x7f1bb4031400 comp rx=0 tx=0).stop 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 msgr2=0x7f1bb819a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cad0 tx=0x7f1bb000cf90 comp rx=0 tx=0).stop 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 shutdown_connections 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1ba40a3d20 0x7f1ba40a61e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb8103400 0x7f1bb819a830 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb8102a60 0x7f1bb81a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 --2- 172.21.15.32:0/2811806495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb8101180 0x7f1bb81a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.741+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 >> 172.21.15.32:0/2811806495 conn(0x7f1bb8077f70 msgr2=0x7f1bb8104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.742+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 shutdown_connections 2024-04-23T18:32:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.742+0000 7f1bc04bb640 1 -- 172.21.15.32:0/2811806495 wait complete. 2024-04-23T18:32:19.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-23T18:32:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.854+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1614656801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac102b50 msgr2=0x7f9eac106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.854+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1614656801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac102b50 0x7f9eac106b90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009900 tx=0x7f9ea002a410 comp rx=0 tx=0).stop 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1614656801 shutdown_connections 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1614656801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac107270 0x7f9eac10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1614656801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac102b50 0x7f9eac106b90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1614656801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1614656801 >> 172.21.15.32:0/1614656801 conn(0x7f9eac0fda40 msgr2=0x7f9eac0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1614656801 shutdown_connections 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1614656801 wait complete. 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 Processor -- start 2024-04-23T18:32:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.855+0000 7f9eb23fe640 1 -- start start 2024-04-23T18:32:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 0x7f9eac19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac112190 con 0x7f9eac107270 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac112010 con 0x7f9eac1021a0 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac112310 con 0x7f9eac19a6d0 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58392/0 (socket says 172.21.15.32:58392) 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 -- 172.21.15.32:0/1713591937 learned_addr learned my addr 172.21.15.32:0/1713591937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb099b640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 0x7f9eac19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eabfff640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 -- 172.21.15.32:0/1713591937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 msgr2=0x7f9eac19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 0x7f9eac19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 -- 172.21.15.32:0/1713591937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 msgr2=0x7f9eac1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 -- 172.21.15.32:0/1713591937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac19b2d0 con 0x7f9eac107270 2024-04-23T18:32:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb099b640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 0x7f9eac19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eabfff640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eab7fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0032fd0 tx=0x7f9ea0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea003f030 con 0x7f9eac107270 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac19b560 con 0x7f9eac107270 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea0004490 con 0x7f9eac107270 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac1a76f0 con 0x7f9eac107270 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.856+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0049060 con 0x7f9eac107270 2024-04-23T18:32:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.858+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac103670 con 0x7f9eac107270 2024-04-23T18:32:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.859+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9ea0042030 con 0x7f9eac107270 2024-04-23T18:32:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.859+0000 7f9ea97fa640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 0x7f9e940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.859+0000 7f9eabfff640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 0x7f9e940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.859+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f9ea00f1120 con 0x7f9eac107270 2024-04-23T18:32:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.860+0000 7f9eabfff640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 0x7f9e940a61e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00abe0 tx=0x7f9e9c00a5c0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:19.863+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ea00bd7e0 con 0x7f9eac107270 2024-04-23T18:32:20.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.101+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f9eac102bf0 con 0x7f9eac107270 2024-04-23T18:32:20.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.110+0000 7f9ea97fa640 1 -- 172.21.15.32:0/1713591937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+746063 (secure 0 0 0) 0x7f9ea00bd0e0 con 0x7f9eac107270 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 25, 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: "active_gid": 15294, 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: "active_name": "x", 2024-04-23T18:32:20.112 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addrs": { 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3410706145 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3410706145 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addr": "172.21.15.32:6801/3410706145", 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "active_change": "2024-04-23T18:15:07.361537+0000", 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "active_mgr_features": 4540701547738038271, 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "available": true, 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "standbys": [ 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 24976, 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "z", 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-23T18:32:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.118 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.119 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:32:20.121 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.127 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.128 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.129 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:20.131 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-23T18:32:20.132 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.133 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.134 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:32:20.135 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:20.136 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.137 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.138 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.142 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-23T18:32:20.143 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.144 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:32:20.146 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.147 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.148 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.149 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.150 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.158 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-23T18:32:20.159 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-23T18:32:20.160 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-23T18:32:20.161 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-23T18:32:20.162 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.163 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.165 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.166 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.167 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.169 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.170 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.171 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.174 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.177 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:32:20.178 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.179 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.190 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.191 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:32:20.192 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:32:20.193 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.196 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-23T18:32:20.197 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-23T18:32:20.198 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-23T18:32:20.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-23T18:32:20.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-23T18:32:20.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.203 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.204 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-23T18:32:20.205 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.206 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-23T18:32:20.207 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-23T18:32:20.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.209 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.210 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.212 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.213 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.221 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.222 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.223 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.226 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.227 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.235 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.236 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.237 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.238 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.239 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.240 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.246 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.252 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.261 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.262 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.264 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.265 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.266 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.267 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.268 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:20.269 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.270 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.271 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.272 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.273 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.274 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.276 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.277 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-23T18:32:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:20.282 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.283 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-23T18:32:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-23T18:32:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-23T18:32:20.292 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-23T18:32:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-23T18:32:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.297 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.298 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.299 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.300 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.303 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.305 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.306 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-23T18:32:20.307 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.308 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.309 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.310 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.311 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.317 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-23T18:32:20.318 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.319 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.320 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.322 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.323 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.329 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.334 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.335 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.336 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.343 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.344 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.345 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.346 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.347 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-23T18:32:20.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.349 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.350 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.353 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-23T18:32:20.353 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.353 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.354 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.361 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.362 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.363 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.372 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.373 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.374 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.375 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.376 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-23T18:32:20.377 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.378 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.379 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.380 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.381 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.385 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.386 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.392 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.393 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.395 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.399 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.402 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.403 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.404 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-23T18:32:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.407 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.413 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.414 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-23T18:32:20.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:32:20.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-23T18:32:20.417 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:32:20.420 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.421 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.422 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.425 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.432 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.433 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:32:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.480 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.481 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.482 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.483 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.484 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.485 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.486 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.488 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.489 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.491 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.492 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.493 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.499 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-23T18:32:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-23T18:32:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-23T18:32:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.507 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.510 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.511 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.512 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.513 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.514 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.519 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.520 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.521 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.522 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.523 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.524 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.526 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-23T18:32:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.534 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-23T18:32:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:32:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:32:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.539 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.542 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 24983, 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "y", 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:32:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.552 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.553 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.554 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.555 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.556 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.557 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-23T18:32:20.558 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.559 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.560 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.561 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.562 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.563 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.564 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-23T18:32:20.565 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.566 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-23T18:32:20.567 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.569 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.570 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.572 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.573 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:32:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-23T18:32:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-23T18:32:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-23T18:32:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-23T18:32:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-23T18:32:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:32:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:32:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:32:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-23T18:32:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:32:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:32:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:32:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.675 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.676 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.683 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.684 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.692 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.695 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-23T18:32:20.696 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.697 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.698 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.699 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.700 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.701 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.702 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.703 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.704 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.705 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-23T18:32:20.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-23T18:32:20.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-23T18:32:20.708 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:20.709 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.710 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.711 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.714 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-23T18:32:20.715 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.716 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.718 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-23T18:32:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.727 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.728 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:32:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:32:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.741 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.742 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.745 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.748 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.749 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-23T18:32:20.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.751 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.753 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.754 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.755 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:32:20.758 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-23T18:32:20.759 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.760 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.761 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.762 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.763 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.764 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.765 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.766 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.767 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.768 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.770 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.771 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.772 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.773 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.776 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-23T18:32:20.777 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.780 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.781 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.783 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:32:20.784 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.785 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.790 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.791 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.793 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-23T18:32:20.794 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.796 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.798 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.799 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.802 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.803 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.805 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.807 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.808 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.810 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.811 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-23T18:32:20.813 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.815 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.816 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-23T18:32:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.824 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.825 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.826 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.827 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.828 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.830 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.831 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.833 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.846 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.847 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-23T18:32:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:20.849 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-23T18:32:20.850 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.851 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:32:20.852 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.853 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.854 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.855 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.856 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.857 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.858 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.859 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.860 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.861 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.862 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.863 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:20.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.865 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.866 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.867 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.868 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.869 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.870 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.871 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.873 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-23T18:32:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.876 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.877 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:20.884 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.888 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.895 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:32:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.899 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-23T18:32:20.903 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-23T18:32:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.907 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.908 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.909 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.910 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.912 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.913 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.914 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.915 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-23T18:32:20.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-23T18:32:20.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.918 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.919 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.920 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.921 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.922 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.924 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.925 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.927 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.929 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.930 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.931 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.932 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.933 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.934 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.935 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.936 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.938 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.940 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:32:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:32:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:32:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.948 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.949 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.950 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.951 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.952 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.953 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.954 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.956 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.957 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.958 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.959 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.960 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-23T18:32:20.962 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-23T18:32:20.963 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.965 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.966 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.967 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.970 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.973 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.976 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.977 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-23T18:32:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:20.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.981 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.984 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.985 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.986 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.987 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.989 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.991 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.993 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "modules": [ 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "dashboard", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "iostat", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "nfs", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "restful", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_schedule" 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-23T18:32:20.995 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.996 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:20.997 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.998 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-23T18:32:21.001 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.002 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.003 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.005 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.020 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.021 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.023 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.024 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-23T18:32:21.025 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.028 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.029 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.030 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.035 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-23T18:32:21.037 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.038 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.039 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.040 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.041 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-23T18:32:21.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:32:21.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-23T18:32:21.044 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-23T18:32:21.045 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.049 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.051 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:21.052 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.053 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.054 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.055 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.056 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:32:21.057 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.058 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.059 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-23T18:32:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-23T18:32:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-23T18:32:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.070 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-23T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-23T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-23T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.080 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-23T18:32:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.083 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.084 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.085 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.086 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:32:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:32:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:32:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:32:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:32:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:32:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-23T18:32:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-23T18:32:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-23T18:32:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-23T18:32:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-23T18:32:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:32:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:32:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:32:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:32:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-23T18:32:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-23T18:32:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-23T18:32:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-23T18:32:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.315 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.318 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.320 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.321 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.323 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.324 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.325 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.326 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.329 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.340 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.341 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.346 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.349 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.350 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.352 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.353 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.354 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.355 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.357 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.359 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.360 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.362 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.363 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.365 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.366 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.367 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.368 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.369 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.370 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.371 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.372 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.374 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.377 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.378 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.379 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.380 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.382 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.383 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.384 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.385 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:32:21.386 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-23T18:32:21.392 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.395 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.397 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.398 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.399 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.400 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.401 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.402 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.404 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.406 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.409 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.410 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.411 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.413 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.414 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.415 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-23T18:32:21.416 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-23T18:32:21.417 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-23T18:32:21.419 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-23T18:32:21.420 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.421 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.422 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-23T18:32:21.423 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-23T18:32:21.424 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-23T18:32:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "services": { 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "dashboard": "https://172.21.15.32:8443/" 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "always_on_modules": { 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "octopus": [ 2024-04-23T18:32:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "pacific": [ 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-23T18:32:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "quincy": [ 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-23T18:32:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "reef": [ 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-23T18:32:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "squid": [ 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-23T18:32:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "last_failure_osd_epoch": 89, 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "active_clients": [ 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2578657470 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "libcephsqlite", 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1943186426 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2817925687 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1999381947 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2250201169 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-23T18:32:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1858952080 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-23T18:32:21.437 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:20 smithi064 ceph-mon[33062]: from='client.17463 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:20 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/785613559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:20 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2811806495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:20 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1713591937' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:20 smithi040 ceph-mon[32626]: from='client.17463 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:21.438 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:20 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/785613559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.439 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:20 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2811806495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.439 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:20 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1713591937' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:32:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 msgr2=0x7f9e940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 0x7f9e940a61e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00abe0 tx=0x7f9e9c00a5c0 comp rx=0 tx=0).stop 2024-04-23T18:32:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 msgr2=0x7f9eac1a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0032fd0 tx=0x7f9ea0004270 comp rx=0 tx=0).stop 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 shutdown_connections 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9eac19a6d0 0x7f9eac19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f9e940a3d20 0x7f9e940a61e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107270 0x7f9eac1a0b50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 --2- 172.21.15.32:0/1713591937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9eac1021a0 0x7f9eac1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 >> 172.21.15.32:0/1713591937 conn(0x7f9eac0fda40 msgr2=0x7f9eac0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 shutdown_connections 2024-04-23T18:32:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:20.115+0000 7f9eb23fe640 1 -- 172.21.15.32:0/1713591937 wait complete. 2024-04-23T18:32:21.445 DEBUG:tasks.check_counter:Getting stats from a 2024-04-23T18:32:21.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-23T18:32:21.446 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:20 smithi032 ceph-mon[29221]: from='client.17463 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:32:21.446 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:20 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/785613559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.446 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:20 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2811806495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:21.446 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:20 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1713591937' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:32:21.472 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:21 smithi032 ceph-mon[29221]: pgmap v543: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 33 op/s 2024-04-23T18:32:21.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:21 smithi040 ceph-mon[32626]: pgmap v543: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 33 op/s 2024-04-23T18:32:21.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:21 smithi064 ceph-mon[33062]: pgmap v543: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 33 op/s 2024-04-23T18:32:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:32:23.486 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:23 smithi032 ceph-mon[29221]: pgmap v544: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 30 op/s 2024-04-23T18:32:23.487 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:23 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:23.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:23 smithi040 ceph-mon[32626]: pgmap v544: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 30 op/s 2024-04-23T18:32:23.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:23 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:23.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:23 smithi064 ceph-mon[33062]: pgmap v544: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 30 op/s 2024-04-23T18:32:23.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:23 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:23.753 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 320, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 310, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1479184, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 191760, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.061190893, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.020137970, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.036871129, 2024-04-23T18:32:23.754 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.008586242, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 309, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.020922296, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000067709 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1479184, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 191760 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.755 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 3, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 239, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 276304, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.017917109, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.012561666, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.002364602, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 236, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.028159371, 2024-04-23T18:32:23.756 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000119319 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 276304 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 4, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 4, 2024-04-23T18:32:23.757 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 2944, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 448, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.000906444, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000048246, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.000861987, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000837642, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000279214 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:23.758 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:23.759 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:23.760 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:23.761 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 149745, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 31, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:23.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:23.763 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 323, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1434035, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 323, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 323, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1434035, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:23.764 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:23.765 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:23.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:23.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:23.765 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:24.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:24 smithi032 ceph-mon[29221]: pgmap v545: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 28 op/s 2024-04-23T18:32:24.435 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-23T18:32:24.435 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-23T18:32:24.435 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-23T18:32:24.435 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:32:24.435 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:32:24.435 DEBUG:tasks.check_counter:Getting stats from d 2024-04-23T18:32:24.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-23T18:32:24.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:24 smithi040 ceph-mon[32626]: pgmap v545: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 28 op/s 2024-04-23T18:32:24.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:24 smithi064 ceph-mon[33062]: pgmap v545: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 28 op/s 2024-04-23T18:32:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:32:26.427 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:26.427 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:26.427 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 331, 2024-04-23T18:32:26.427 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 319, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1484096, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 196208, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.063017402, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.022996091, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.038262578, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.007506274, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 319, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.017053210, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000053458 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:26.428 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1484096, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 196208 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 3, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 243, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 277008, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.019975156, 2024-04-23T18:32:26.429 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.014022113, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.002399814, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 240, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.022423077, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000093429 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.430 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 277008 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 0, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 2, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 0, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.000302716, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.000205883, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:26.431 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-23T18:32:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:26.433 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 148714, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 30, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-23T18:32:26.435 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:26.436 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 330, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1435009, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 330, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 330, 2024-04-23T18:32:26.437 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1435009, 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:26.438 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:27.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:26 smithi032 ceph-mon[29221]: pgmap v546: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 135 KiB/s wr, 32 op/s 2024-04-23T18:32:27.039 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-23T18:32:27.039 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-23T18:32:27.039 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-23T18:32:27.039 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:32:27.039 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:32:27.039 DEBUG:tasks.check_counter:Getting stats from g 2024-04-23T18:32:27.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-23T18:32:27.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:26 smithi040 ceph-mon[32626]: pgmap v546: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 135 KiB/s wr, 32 op/s 2024-04-23T18:32:27.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:26 smithi064 ceph-mon[33062]: pgmap v546: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 135 KiB/s wr, 32 op/s 2024-04-23T18:32:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:32:28.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:28 smithi032 ceph-mon[29221]: pgmap v547: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 134 KiB/s wr, 30 op/s 2024-04-23T18:32:28.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:28 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:29.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:28 smithi040 ceph-mon[32626]: pgmap v547: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 134 KiB/s wr, 30 op/s 2024-04-23T18:32:29.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:28 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:29.151 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:29.151 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:29.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 4, 2024-04-23T18:32:29.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 4, 2024-04-23T18:32:29.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 2944, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 448, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.001813870, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000044214, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.000731888, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000800384, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000266794 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:29.152 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 3, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 241, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 276656, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.018547310, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.012998643, 2024-04-23T18:32:29.153 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.002336350, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 238, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.020683002, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000086903 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 276656 2024-04-23T18:32:29.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 322, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 312, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1479728, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 193040, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.056882669, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.019742277, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.036953594, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.007215136, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 311, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.014130899, 2024-04-23T18:32:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000045436 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1479728, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 193040 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-23T18:32:29.156 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:29.157 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:29.158 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 148754, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 31, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-23T18:32:29.159 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:29.160 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 325, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1434295, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 325, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-23T18:32:29.161 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 325, 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1434295, 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:29.162 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:28 smithi064 ceph-mon[33062]: pgmap v547: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 134 KiB/s wr, 30 op/s 2024-04-23T18:32:29.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:28 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:29.889 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-23T18:32:29.889 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-23T18:32:29.889 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-23T18:32:29.889 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:32:29.889 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:32:29.890 DEBUG:tasks.check_counter:Getting stats from j 2024-04-23T18:32:29.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-23T18:32:30.142 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:32:31.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:30 smithi032 ceph-mon[29221]: pgmap v548: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 164 KiB/s wr, 36 op/s 2024-04-23T18:32:31.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:30 smithi040 ceph-mon[32626]: pgmap v548: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 164 KiB/s wr, 36 op/s 2024-04-23T18:32:31.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:30 smithi064 ceph-mon[33062]: pgmap v548: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 164 KiB/s wr, 36 op/s 2024-04-23T18:32:31.886 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 333, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 321, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1484640, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 197488, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.056161498, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.018889523, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.034493377, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.008438894, 2024-04-23T18:32:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 321, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.030454778, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000094874 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1484640, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 197488 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 3, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 244, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 277184, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.019575708, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.013971402, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.002212740, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 241, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.028199579, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000117010 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 277184 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 0, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 2, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 0, 2024-04-23T18:32:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.000317813, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.000244057, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 148731, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 31, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-23T18:32:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 332, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1435269, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 332, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 332, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1435269, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-23T18:32:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-23T18:32:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-23T18:32:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-23T18:32:31.898 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-23T18:32:32.586 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-23T18:32:32.586 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-23T18:32:32.586 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-23T18:32:32.586 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:32:32.586 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:32:32.587 DEBUG:tasks.check_counter:Getting stats from b 2024-04-23T18:32:32.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-23T18:32:32.849 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:32:32.865 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:32 smithi040 ceph-mon[32626]: pgmap v549: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:33.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:32 smithi032 ceph-mon[29221]: pgmap v549: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:33.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:32 smithi064 ceph-mon[33062]: pgmap v549: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:34.823 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 7364, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 7406, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2672032, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 19232405, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 203, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.650161768, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.240531373, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.303939569, 2024-04-23T18:32:34.824 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.109647911, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 7407, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.433243324, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000193498 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8596, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002068975, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000240 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 42224, 2024-04-23T18:32:34.825 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 976 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2392, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2424, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2323243, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 9010832, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 15, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 6, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.229021832, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.077186592, 2024-04-23T18:32:34.826 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.085173470, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.065762302, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2422, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.672242824, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000277556 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 292, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000058376, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000199 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1773872, 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 711536 2024-04-23T18:32:34.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 12664, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 15362, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 5124520, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 13569622, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 21, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 8, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 1.066705500, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.387741483, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.648531246, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.183437574, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 15358, 2024-04-23T18:32:34.828 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.076858222, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000070117 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 23270, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.004271371, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000183 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3074512, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 5557440 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-23T18:32:34.829 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4688, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.648482555, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000778259 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:34.830 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 571, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.024170047, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000042329 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 2916, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 2669, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2669, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 67.639637051, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.025342689 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.831 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 247, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 1, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 806, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 5, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 571, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 105, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 456, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 10, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 422, 2024-04-23T18:32:34.832 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 1436, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 332, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 332, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 23, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 3882, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 3054, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 246, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 5, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 1, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 38, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 15170, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 27, 2024-04-23T18:32:34.833 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 1392, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 18, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 805, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 1009, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 1417173128, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:34.834 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 24, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 1566, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 2330, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 2926, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 1183, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 174, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 497 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.835 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 5, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 39, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 9, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 200, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 171, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 20, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:34.836 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 14, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 2178, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 27, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 5, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 4844, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 4724, 2024-04-23T18:32:34.837 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 4724, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 120, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 47, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 79, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 78, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 78, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 1, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 1, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 1, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 26079217, 2024-04-23T18:32:34.838 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 26738567, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2280, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 8.668803618, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003802106 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 538, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 2065, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 1527, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 67, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 118, 2024-04-23T18:32:34.839 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 51, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 574, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 2483, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 1909, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 332, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 1674, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 1342, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 76196, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 231800 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 3442, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 892, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 2916, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 239, 2024-04-23T18:32:34.840 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 1885, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1157, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 64.551107571, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.055791795 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 420, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.606108386, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001443115 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 23, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.092808544, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004035154 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 246, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.954122521, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003878546 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.842 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000147162, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000147162 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:34.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-23T18:32:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 17, 2024-04-23T18:32:34.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002712068, 2024-04-23T18:32:34.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000159533 2024-04-23T18:32:34.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.846 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-23T18:32:34.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 37, 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.254329614, 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006873773 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 17, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.084131553, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004948914 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 449, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.172806860, 2024-04-23T18:32:34.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000384870 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 52, 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.094981429, 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001826565 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-23T18:32:34.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 190, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.624044786, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003284446 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.850 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-23T18:32:34.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011304885, 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005652442 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 11, 2024-04-23T18:32:34.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.052194894, 2024-04-23T18:32:34.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004744990 2024-04-23T18:32:34.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.853 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:34.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001010942, 2024-04-23T18:32:34.854 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001010942 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.856 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-23T18:32:34.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 7, 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.017588233, 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002512604 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 39, 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.120237603, 2024-04-23T18:32:34.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003083015 2024-04-23T18:32:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-04-23T18:32:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 5, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 3, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 135, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 67, 2024-04-23T18:32:34.862 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 1017, 2024-04-23T18:32:34.863 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:34.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.863 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-23T18:32:34.863 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 360, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 360, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:34.864 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:34.866 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:34.867 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:34.868 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:34.869 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:34.870 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 514921, 2024-04-23T18:32:34.870 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 686, 2024-04-23T18:32:34.870 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:34.870 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:34.870 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:34.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-23T18:32:34.872 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:34.872 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:34.872 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:34.872 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-23T18:32:34.872 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 2023067, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 30751, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.873 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-23T18:32:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-23T18:32:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-23T18:32:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 5811, 2024-04-23T18:32:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 27334062, 2024-04-23T18:32:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 34, 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 5811, 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5811, 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 112.064414586, 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.019284876 2024-04-23T18:32:34.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5811, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10037 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 5811, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 70, 2024-04-23T18:32:34.876 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 5741, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 719, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 1261, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 69, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 2349, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 346, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-23T18:32:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 4, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 708, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2024-04-23T18:32:34.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 2522, 2024-04-23T18:32:34.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:34.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:34.881 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:34.881 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:34.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-23T18:32:34.883 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 113, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 28, 2024-04-23T18:32:34.884 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 1911, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 145 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-23T18:32:34.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 2551 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 17, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 10, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 185, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 171, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.886 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 22416, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 7678873, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 22416, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 22416, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 7678873, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.887 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 5811, 2024-04-23T18:32:34.888 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 27351597, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5811, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 27351597, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:34.889 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 5811, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 5811, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5811, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 5811, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 171, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 17551, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 171, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 69, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 17551, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.891 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x560c49f40460": { 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 4844, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 22544263, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 4844, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:34.892 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2280, 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 22544263, 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:34.893 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-23T18:32:35.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:34 smithi032 ceph-mon[29221]: pgmap v550: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:35.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:34 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:35.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:34 smithi040 ceph-mon[32626]: pgmap v550: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:35.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:34 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:35.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:34 smithi064 ceph-mon[33062]: pgmap v550: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 11 op/s 2024-04-23T18:32:35.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:34 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:35.775 INFO:tasks.check_counter:Daemon mds.b mds.exported=27 2024-04-23T18:32:35.775 INFO:tasks.check_counter:Daemon mds.b mds.imported=18 2024-04-23T18:32:35.775 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-23T18:32:35.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=17 2024-04-23T18:32:35.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=11 2024-04-23T18:32:35.776 DEBUG:tasks.check_counter:Getting stats from e 2024-04-23T18:32:35.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-23T18:32:36.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:35 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:36.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:35 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:36.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:35 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:32:36.026 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:32:36.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:35 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:36.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:35 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:36.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:35 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:32:36.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:35 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:36.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:35 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:36.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:35 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' 2024-04-23T18:32:37.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:37 smithi040 ceph-mon[32626]: pgmap v551: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 13 op/s 2024-04-23T18:32:37.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:37 smithi032 ceph-mon[29221]: pgmap v551: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 13 op/s 2024-04-23T18:32:37.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:37 smithi064 ceph-mon[33062]: pgmap v551: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 13 op/s 2024-04-23T18:32:37.929 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1563, 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1517, 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2052800, 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 573224, 2024-04-23T18:32:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.181684529, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.064455842, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.070415011, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.043274413, 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1517, 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.094185163, 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000062086 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1661696, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 203072 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 156, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 397, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 312825, 2024-04-23T18:32:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 728652, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.027172008, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.011886346, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.006612542, 2024-04-23T18:32:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.005108872, 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 394, 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.026170866, 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000066423 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 679936 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2419, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2421, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 13197479, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 769249, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.162680295, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.034825662, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.053939975, 2024-04-23T18:32:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.064044098, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2419, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.074401803, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000030757 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:32:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6271, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.147588981, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000023535 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 826, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 169, 2024-04-23T18:32:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 657, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 584, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-04-23T18:32:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 7, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-23T18:32:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-04-23T18:32:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-23T18:32:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-23T18:32:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-23T18:32:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 3085, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 7, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-23T18:32:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 17051523, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 17050949, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 3085 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-23T18:32:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 407, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 1136, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 729, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 43, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 63, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 20, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 826, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 1643, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 817, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 58068, 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215416 2024-04-23T18:32:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-23T18:32:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-04-23T18:32:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-04-23T18:32:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-04-23T18:32:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-04-23T18:32:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 4, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 2, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 3, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-04-23T18:32:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 2894, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 212619, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 660, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-23T18:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 1796693, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 40971, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-23T18:32:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 3736, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 21, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 3, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 3736, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3736, 2024-04-23T18:32:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 4.690935655, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001255603 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3736, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3737 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 3736, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 3736, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 2252, 2024-04-23T18:32:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 1482, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2024-04-23T18:32:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-23T18:32:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-23T18:32:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-23T18:32:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-23T18:32:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-23T18:32:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-23T18:32:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 113, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 27, 2024-04-23T18:32:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 5, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 6, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 1, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-04-23T18:32:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-23T18:32:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:32:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 4134, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 15167991, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 4134, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 4134, 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 15167991, 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3736, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 12857224, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3736, 2024-04-23T18:32:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 12857224, 2024-04-23T18:32:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3736, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 3736, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3736, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 3736, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x5652ca8ec6e0": { 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:37.993 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-23T18:32:38.664 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-23T18:32:38.664 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-23T18:32:38.664 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-23T18:32:38.664 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:32:38.664 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:32:38.664 DEBUG:tasks.check_counter:Getting stats from h 2024-04-23T18:32:38.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-23T18:32:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:32:39.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:39 smithi040 ceph-mon[32626]: pgmap v552: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-04-23T18:32:39.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:39 smithi032 ceph-mon[29221]: pgmap v552: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-04-23T18:32:39.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:39 smithi064 ceph-mon[33062]: pgmap v552: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 6240, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 6321, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2568994, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 15437674, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 10, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.506652592, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.248100507, 2024-04-23T18:32:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.228840725, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.051347430, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6320, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.281214093, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000044495 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8228, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001766643, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000214 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 466272 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4475, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 6392, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1554345, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 3701762, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 20, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 9, 2024-04-23T18:32:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.350213946, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.154042976, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.224852468, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.002188671, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6387, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.315486601, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000049395 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 9196, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001520217, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000165 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1449552, 2024-04-23T18:32:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 3587104 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1869, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1753, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2501054, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1464617, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.160293473, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.055692334, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.101286503, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.021358951, 2024-04-23T18:32:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1754, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.070282892, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000040070 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1952, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000332608, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000170 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1671648, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 208048 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3338, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.786558145, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000535218 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-23T18:32:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 280, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.013990761, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000049967 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 1927, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 1880, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1880, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 137.234013574, 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.072996815 2024-04-23T18:32:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 47, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 17, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 19, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 1, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 687, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 84, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 448, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 155, 2024-04-23T18:32:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 567, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 800, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 173, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 173, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 17, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 4918, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 3713, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 36, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 27, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 1, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 324, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 3633, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 18, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 706, 2024-04-23T18:32:40.855 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 20, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 838, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 1027, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 1657943348, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:40.856 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 10, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 1088, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 1653, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 1966, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 615, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 102, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 962 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 30, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 10, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 2, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 256, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 81, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 173, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:40.858 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 14, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 2231, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 18, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 1, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.859 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 3085, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 3085, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 7, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 7, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-04-23T18:32:40.860 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 17051523, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1733, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 11.916731974, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006876360 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 399, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 1151, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 752, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 69, 2024-04-23T18:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 84, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 15, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 699, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 1672, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 973, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 173, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 793, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 620, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 63764, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223608 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 3355, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 99, 2024-04-23T18:32:40.862 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 1927, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 418, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 1023, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 131, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 121.210046729, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.925267532 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 118, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10.921064895, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.092551397 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:40.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 185, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.508825143, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002750406 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 737, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.934117387, 2024-04-23T18:32:40.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001267459 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-23T18:32:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.868 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 234, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.174257437, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005018194 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 43, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.038531045, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000896070 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.869 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 234, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.054654639, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008780575 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 198, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.392516299, 2024-04-23T18:32:40.875 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001982405 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 4, 2024-04-23T18:32:40.876 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 2, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 241, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 120, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 1025, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:40.878 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:40.879 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:40.880 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 724108, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 709, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 88, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 1, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 1881745, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 50577, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 2309, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 13053500, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 28, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 2309, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2309, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 15.029793907, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006509222 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2309, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2655 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 2309, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 51, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 2258, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-23T18:32:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 10, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 34, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 1767, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 98, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 1, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 96, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-23T18:32:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 2, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 20, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-23T18:32:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 113, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 27, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2024-04-23T18:32:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 494, 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 34, 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 99 2024-04-23T18:32:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 2292 2024-04-23T18:32:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 10, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 81, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 81, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 12580, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 5275687, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 12580, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 12580, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 5275687, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2309, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 13062105, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2309, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 13062105, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2309, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2309, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2309, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2309, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 81, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 8877, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 81, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 34, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 8877, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.897 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55f59b56e460": { 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3085, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 12857219, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3085, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:40.898 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1733, 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 12857219, 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:40.899 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-23T18:32:41.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:41 smithi032 ceph-mon[29221]: pgmap v553: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 11 op/s 2024-04-23T18:32:41.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:41 smithi040 ceph-mon[32626]: pgmap v553: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 11 op/s 2024-04-23T18:32:41.663 INFO:tasks.check_counter:Daemon mds.h mds.exported=18 2024-04-23T18:32:41.663 INFO:tasks.check_counter:Daemon mds.h mds.imported=20 2024-04-23T18:32:41.663 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-23T18:32:41.663 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:32:41.664 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:32:41.664 DEBUG:tasks.check_counter:Getting stats from k 2024-04-23T18:32:41.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-23T18:32:41.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:41 smithi064 ceph-mon[33062]: pgmap v553: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 11 op/s 2024-04-23T18:32:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:32:43.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:43 smithi040 ceph-mon[32626]: pgmap v554: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:32:43.472 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:43 smithi064 ceph-mon[33062]: pgmap v554: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:32:43.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:43 smithi032 ceph-mon[29221]: pgmap v554: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 333, 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 323, 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1482784, 2024-04-23T18:32:43.728 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 199600, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.060396815, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.018192523, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.040899836, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.008145771, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 322, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.015943827, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000049514 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:43.729 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1482784, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 199600 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 3, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 248, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 277888, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.015175961, 2024-04-23T18:32:43.730 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010058490, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.002284512, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 245, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.023538362, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000096074 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 277888 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2944, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.000653160, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000031393, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000575218, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:43.732 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000743035, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000247678 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-23T18:32:43.733 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:43.734 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:43.735 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 149745, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:43.736 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-23T18:32:43.737 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-23T18:32:43.738 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 336, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1435789, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 336, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 336, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1435789, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:43.739 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-23T18:32:43.740 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-23T18:32:44.440 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-23T18:32:44.441 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-23T18:32:44.441 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-23T18:32:44.441 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:32:44.441 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:32:44.441 DEBUG:tasks.check_counter:Getting stats from c 2024-04-23T18:32:44.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-23T18:32:44.705 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:32:45.404 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:45 smithi064 ceph-mon[33062]: pgmap v555: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:32:45.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:45 smithi032 ceph-mon[29221]: pgmap v555: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:32:45.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:45 smithi040 ceph-mon[32626]: pgmap v555: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 291, 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 284, 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 277646, 2024-04-23T18:32:46.833 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 105349, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 2, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.036730304, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.011236409, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.013125833, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.009684389, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 283, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.020500684, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000072440 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:46.834 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 202176, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 37920 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 258, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 302, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1520088, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 244759, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-04-23T18:32:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.026901632, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.008781202, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.009096892, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.006288569, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 300, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.014288825, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000047629 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-23T18:32:46.836 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 162976 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 216, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 216, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 55700, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 67871, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.021114666, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.007001796, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.005453277, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.006793780, 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 215, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.009558614, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000044458 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.838 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1138, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.035017426, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000030771 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:46.839 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 0, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 0, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-04-23T18:32:46.840 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 421, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 154, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 267, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 201, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 0, 2024-04-23T18:32:46.841 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 19, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 0, 2024-04-23T18:32:46.842 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 1009, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 1417173128, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 6, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:46.843 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 0 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-04-23T18:32:46.844 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:32:46.845 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 266, 2024-04-23T18:32:46.846 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 2, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 9, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 26079217, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 26738567, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 26713012, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 266 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 358, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 358, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 0, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 26, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 27, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 1, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 421, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 421, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 0, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 0, 2024-04-23T18:32:46.848 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 53916, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 215416 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.850 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.851 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.852 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-04-23T18:32:46.853 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-04-23T18:32:46.855 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-04-23T18:32:46.856 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:46.857 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.859 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:46.860 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 2, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 2, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 0, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 2280, 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:46.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:46.862 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 221511, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 492, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-23T18:32:46.864 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 165, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 1260004, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 21813, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:46.865 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 681, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 26, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 4, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 681, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 681, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.981981391, 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001441969 2024-04-23T18:32:46.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 681, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 682 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 681, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 681, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 0, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 418, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 0, 2024-04-23T18:32:46.867 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 260, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 2, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 0, 2024-04-23T18:32:46.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-04-23T18:32:46.869 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 113, 2024-04-23T18:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 6, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 7, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 7, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 2, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 0 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-04-23T18:32:46.871 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 0, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:46.872 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 761, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1778006, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 761, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 761, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1778006, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 681, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 1452567, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 681, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1452567, 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:46.874 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 681, 2024-04-23T18:32:46.875 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 681, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 681, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 681, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:46.876 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x556ffe8f06e0": { 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:46.877 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:46.878 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-23T18:32:47.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:47 smithi032 ceph-mon[29221]: pgmap v556: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:32:47.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:47 smithi040 ceph-mon[32626]: pgmap v556: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:32:47.670 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:47 smithi064 ceph-mon[33062]: pgmap v556: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:32:47.700 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-23T18:32:47.700 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-23T18:32:47.700 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-23T18:32:47.700 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:32:47.700 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:32:47.700 DEBUG:tasks.check_counter:Getting stats from f 2024-04-23T18:32:47.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-23T18:32:47.954 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:32:49.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:49 smithi032 ceph-mon[29221]: pgmap v557: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:32:49.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:49 smithi064 ceph-mon[33062]: pgmap v557: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:32:49.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:49 smithi040 ceph-mon[32626]: pgmap v557: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 885, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 887, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 4462950, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 281494, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 7, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 2, 2024-04-23T18:32:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.077464450, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.021532234, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.024910905, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.023278966, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 885, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.029628475, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000033478 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 1194, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 1436, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 4496302, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 975837, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 7, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 5, 2024-04-23T18:32:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.121387366, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.037480980, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.032931461, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.037999608, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1434, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.079121181, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000055175 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 598048 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 2000, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 1956, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 2335728, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 713658, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.225018741, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.067502646, 2024-04-23T18:32:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.083149554, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.067109962, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1955, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.094111632, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000048138 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1659296, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 203904 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 6144, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.172682492, 2024-04-23T18:32:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000028105 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 0, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 0, 2024-04-23T18:32:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 787, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 303, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 484, 2024-04-23T18:32:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 353, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 18, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 0, 2024-04-23T18:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 0, 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 0 2024-04-23T18:32:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-04-23T18:32:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 1373, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 4, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 2, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 4194304, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 12759088, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 12740724, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 1373 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 759, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 1334, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 575, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 51, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 57, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 6, 2024-04-23T18:32:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 787, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 1418, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 631, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 0, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 0, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 0, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 57824, 2024-04-23T18:32:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 215416 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-04-23T18:32:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-04-23T18:32:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-04-23T18:32:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-04-23T18:32:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 2, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 3, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 1, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 2, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-04-23T18:32:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 2282, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 229504, 2024-04-23T18:32:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 856, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 165, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-23T18:32:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 2393818, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 22845, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 3676, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 21, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 3, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 3676, 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-04-23T18:32:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3676, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 8.858830600, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.002409910 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3676, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 3677 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 3676, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 3676, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 0, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-04-23T18:32:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 2266, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 1408, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 1, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-04-23T18:32:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 113, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 27, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 6, 2024-04-23T18:32:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 7, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 1, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 0, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 2, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 0 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 0, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-04-23T18:32:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 4075, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 10905116, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 4075, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 4075, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 10905116, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 3676, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 8603134, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 3676, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 8603134, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 3676, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 3676, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 3676, 2024-04-23T18:32:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 3676, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x556511cfa6e0": { 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:49.837 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-23T18:32:50.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:50 smithi032 ceph-mon[29221]: pgmap v558: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-23T18:32:50.615 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:50 smithi040 ceph-mon[32626]: pgmap v558: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-23T18:32:50.665 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:50 smithi064 ceph-mon[33062]: pgmap v558: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-23T18:32:50.700 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-23T18:32:50.700 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-23T18:32:50.700 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-23T18:32:50.700 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:32:50.700 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:32:50.700 DEBUG:tasks.check_counter:Getting stats from i 2024-04-23T18:32:50.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-23T18:32:50.956 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 1586, 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 1667, 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 883139, 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 5065381, 2024-04-23T18:32:52.799 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 14, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.110626428, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.041640115, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.056897731, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.014733552, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1665, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.088651795, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000053244 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1924, 2024-04-23T18:32:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000333217, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000173 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 928, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 736 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 3031, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 5196, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1021104, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 7391682, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 12, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 5, 2024-04-23T18:32:52.801 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.301010686, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.134366403, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.164372339, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.022472095, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 5194, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.416302969, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000080150 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 5735, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.001136902, 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000198 2024-04-23T18:32:52.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 889968, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 3216848 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 4439, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 3646, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3565762, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 2147284, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 8, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 6, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.308823791, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.124201662, 2024-04-23T18:32:52.803 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.209404480, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.015580984, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3645, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.181737435, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000049859 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 6834, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.001662183, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000243 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1670592, 2024-04-23T18:32:52.804 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 218176 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1577, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 1.137077691, 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000721038 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 42, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.001711357, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000040746 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 851, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 823, 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-04-23T18:32:52.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 823, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 24.489159351, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.029755965 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 28, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 4, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 19, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 1, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 777, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 23, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 687, 2024-04-23T18:32:52.807 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 67, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 744, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 644, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 562, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 562, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 22, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 1511, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 1433, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 18, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 12, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 2, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 1011, 2024-04-23T18:32:52.808 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 12, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 612, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 19, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 1067, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 1, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 1027, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 1657943348, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 6, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-04-23T18:32:52.809 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 20, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 1044, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 1656, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 1640, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 338, 2024-04-23T18:32:52.810 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 31, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 74 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 0, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 23, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 9, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 6, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 16, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 9, 2024-04-23T18:32:52.811 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 1, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_path": 14, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_quiesce_inode": 10511, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 12, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 1, 2024-04-23T18:32:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 1373, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 1373, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 4, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-04-23T18:32:52.813 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 4, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 2, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 4194304, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 12759088, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 0, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 809, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 7.996854360, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.009884863 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.814 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 0 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 775, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 1379, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 604, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 68, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 71, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 3, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 779, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 1463, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 684, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 562, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 1144, 2024-04-23T18:32:52.815 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 582, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 66460, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 223608 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 959, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 214, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 851, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 51, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 562, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 8, 2024-04-23T18:32:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.016156108, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.002019513 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 235, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 6.591604346, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.028049380 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.817 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.021460978, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.007153659 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 415, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 17.597838171, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.042404429 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.818 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000097978, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000097978 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-04-23T18:32:52.819 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-04-23T18:32:52.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 107, 2024-04-23T18:32:52.822 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.157991284, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001476554 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 37, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.055447652, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001498585 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.008646226, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.008646226 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.823 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.827 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 16, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.039916608, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.002494788 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 2, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 3, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 1, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 2, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 121, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 60, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 1033, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-23T18:32:52.829 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:52.830 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:52.831 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 483007, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 839, 2024-04-23T18:32:52.832 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 7936, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 165, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-23T18:32:52.833 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 2748539, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 34715, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 1061, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 8682475, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 28, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 1061, 2024-04-23T18:32:52.834 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1061, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 11.005788735, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.010373033 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1061, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 1219 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 1061, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 8, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 1053, 2024-04-23T18:32:52.835 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 10, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 4, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 813, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 71, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 24, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-04-23T18:32:52.836 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 20, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-04-23T18:32:52.837 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-04-23T18:32:52.838 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 113, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 27, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 10, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 11, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 1, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 211, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 8, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 58 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-04-23T18:32:52.839 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 1, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 10695 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 5, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 5, 2024-04-23T18:32:52.840 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 9, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 9, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 9052, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 4518231, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 9052, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:52.841 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 9052, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 4518231, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:52.842 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 1061, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 8683568, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 1061, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 8683568, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-04-23T18:32:52.843 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 1061, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 1061, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 1061, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1061, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.844 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 9, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1157, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 9, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:52.845 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 4, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1157, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x55e43e7ae460": { 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 1373, 2024-04-23T18:32:52.846 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 8564784, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 1373, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 809, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 8564784, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:52.847 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-23T18:32:53.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:52 smithi032 ceph-mon[29221]: pgmap v559: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:53.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:52 smithi040 ceph-mon[32626]: pgmap v559: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:53.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:52 smithi064 ceph-mon[33062]: pgmap v559: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:53.695 INFO:tasks.check_counter:Daemon mds.i mds.exported=12 2024-04-23T18:32:53.695 INFO:tasks.check_counter:Daemon mds.i mds.imported=19 2024-04-23T18:32:53.695 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-23T18:32:53.695 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:32:53.695 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:32:53.696 DEBUG:tasks.check_counter:Getting stats from l 2024-04-23T18:32:53.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid cdb712f8-019b-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-23T18:32:53.951 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:32:55.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:54 smithi032 ceph-mon[29221]: pgmap v560: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-23T18:32:55.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:54 smithi040 ceph-mon[32626]: pgmap v560: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-23T18:32:55.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:54 smithi064 ceph-mon[33062]: pgmap v560: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-23T18:32:55.808 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001398994, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000327776, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 7, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 253, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3424, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 278864, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.016035975, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.010405670, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002438302, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 251, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.029185001, 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000116274 2024-04-23T18:32:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3424, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 278864 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 336, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 326, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1483600, 2024-04-23T18:32:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 201520, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.057672671, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.018269338, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.037892767, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.008393668, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 325, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.018273918, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000056227 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:32:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1483600, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 201520 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:32:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:32:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:32:55.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148754, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 31, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-23T18:32:55.817 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-04-23T18:32:55.818 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 339, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1436179, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 339, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-23T18:32:55.819 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 339, 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1436179, 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-23T18:32:55.820 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-23T18:32:56.549 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-23T18:32:56.549 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-23T18:32:56.549 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-23T18:32:56.549 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:32:56.549 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:32:56.549 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T18:32:56.561 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T18:32:56.561 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:32:56.561 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:32:56.582 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:32:56.582 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:32:56.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 -- 172.21.15.32:0/1365680607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b406a1b0 msgr2=0x7f94b4106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/1365680607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b406a1b0 0x7f94b4106c50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009900 tx=0x7f94a402a480 comp rx=0 tx=0).stop 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 -- 172.21.15.32:0/1365680607 shutdown_connections 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/1365680607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b41008a0 0x7f94b4100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/1365680607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b406a1b0 0x7f94b4106c50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/1365680607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b4069dd0 0x7f94b4106710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:56.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 -- 172.21.15.32:0/1365680607 >> 172.21.15.32:0/1365680607 conn(0x7f94b4076130 msgr2=0x7f94b4076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:56.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 -- 172.21.15.32:0/1365680607 shutdown_connections 2024-04-23T18:32:56.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.808+0000 7f94bb3bf640 1 -- 172.21.15.32:0/1365680607 wait complete. 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 Processor -- start 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 -- start start 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4069dd0 0x7f94b4102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b406a1b0 0x7f94b41027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4111080 con 0x7f94b41008a0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4110f00 con 0x7f94b4069dd0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94bb3bf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4111200 con 0x7f94b406a1b0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94b995c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94b995c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47726/0 (socket says 172.21.15.32:47726) 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.809+0000 7f94b995c640 1 -- 172.21.15.32:0/525863689 learned_addr learned my addr 172.21.15.32:0/525863689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b895a640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b406a1b0 0x7f94b41027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b915b640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4069dd0 0x7f94b4102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 -- 172.21.15.32:0/525863689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b406a1b0 msgr2=0x7f94b41027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b406a1b0 0x7f94b41027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 -- 172.21.15.32:0/525863689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4069dd0 msgr2=0x7f94b4102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4069dd0 0x7f94b4102270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 -- 172.21.15.32:0/525863689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b41a6bb0 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94b995c640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00cad0 tx=0x7f94ac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00ae20 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94ac005020 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b41a6ea0 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00f6b0 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.810+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b41a7370 con 0x7f94b41008a0 2024-04-23T18:32:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.811+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b4069250 con 0x7f94b41008a0 2024-04-23T18:32:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.812+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f94ac0051d0 con 0x7f94b41008a0 2024-04-23T18:32:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.813+0000 7f94aa7fc640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 0x7f94a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.813+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f94ac0d0a70 con 0x7f94b41008a0 2024-04-23T18:32:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.813+0000 7f94b915b640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 0x7f94a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.814+0000 7f94b915b640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 0x7f94a00a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009bd0 tx=0x7f94b0005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:56.817+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94ac014030 con 0x7f94b41008a0 2024-04-23T18:32:56.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.005+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94b41037a0 con 0x7f94b41008a0 2024-04-23T18:32:56.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.007+0000 7f94aa7fc640 1 -- 172.21.15.32:0/525863689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23004 (secure 0 0 0) 0x7f94b41037a0 con 0x7f94b41008a0 2024-04-23T18:32:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:57.000 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:32:10.775144+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:32:57.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 msgr2=0x7f94a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 0x7f94a00a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009bd0 tx=0x7f94b0005f90 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 msgr2=0x7f94b4102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00cad0 tx=0x7f94ac00cf90 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 shutdown_connections 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f94a00a3d20 0x7f94a00a61e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b41008a0 0x7f94b4102cf0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94b406a1b0 0x7f94b41027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 --2- 172.21.15.32:0/525863689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4069dd0 0x7f94b4102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 >> 172.21.15.32:0/525863689 conn(0x7f94b4076130 msgr2=0x7f94b41043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 shutdown_connections 2024-04-23T18:32:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.009+0000 7f94bb3bf640 1 -- 172.21.15.32:0/525863689 wait complete. 2024-04-23T18:32:57.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:56 smithi032 ceph-mon[29221]: pgmap v561: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:32:57.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:08:22.076400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:13:03.563177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:13:04.331221+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:32:57.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:32:57.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:56 smithi040 ceph-mon[32626]: pgmap v561: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.127+0000 7f80848cb640 1 -- 172.21.15.32:0/2246510198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080107250 msgr2=0x7f808010ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.127+0000 7f80848cb640 1 --2- 172.21.15.32:0/2246510198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080107250 0x7f808010ee20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8078009ca0 tx=0x7f807802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 -- 172.21.15.32:0/2246510198 shutdown_connections 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 --2- 172.21.15.32:0/2246510198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080107250 0x7f808010ee20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 --2- 172.21.15.32:0/2246510198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8080102b30 0x7f8080106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 --2- 172.21.15.32:0/2246510198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080102180 0x7f8080102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 -- 172.21.15.32:0/2246510198 >> 172.21.15.32:0/2246510198 conn(0x7f80800fda00 msgr2=0x7f80800ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 -- 172.21.15.32:0/2246510198 shutdown_connections 2024-04-23T18:32:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 -- 172.21.15.32:0/2246510198 wait complete. 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 Processor -- start 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.128+0000 7f80848cb640 1 -- start start 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 0x7f80801a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 0x7f808019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112280 con 0x7f8080102180 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112100 con 0x7f8080107250 2024-04-23T18:32:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112400 con 0x7f808019a7c0 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47748/0 (socket says 172.21.15.32:47748) 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 -- 172.21.15.32:0/1176786594 learned_addr learned my addr 172.21.15.32:0/1176786594 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807dd9b640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 0x7f80801a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807ed9d640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 0x7f808019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 -- 172.21.15.32:0/1176786594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 msgr2=0x7f808019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 0x7f808019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 -- 172.21.15.32:0/1176786594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 msgr2=0x7f80801a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 0x7f80801a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 -- 172.21.15.32:0/1176786594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808019b350 con 0x7f8080102180 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807dd9b640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 0x7f80801a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807ed9d640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 0x7f808019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f807e59c640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f807000b4a0 tx=0x7f807000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070014070 con 0x7f8080102180 2024-04-23T18:32:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808019b640 con 0x7f8080102180 2024-04-23T18:32:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8070010ac0 con 0x7f8080102180 2024-04-23T18:32:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070017c30 con 0x7f8080102180 2024-04-23T18:32:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.129+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a7820 con 0x7f8080102180 2024-04-23T18:32:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.131+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8080103650 con 0x7f8080102180 2024-04-23T18:32:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.132+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8070017070 con 0x7f8080102180 2024-04-23T18:32:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.132+0000 7f806f7fe640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 0x7f80680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.132+0000 7f807dd9b640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 0x7f80680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.132+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f80700cfe70 con 0x7f8080102180 2024-04-23T18:32:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.133+0000 7f807dd9b640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 0x7f80680a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f80740058c0 tx=0x7f8074005850 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.137+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f808019b640 con 0x7f8080102180 2024-04-23T18:32:57.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:56 smithi064 ceph-mon[33062]: pgmap v561: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-23T18:32:57.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.322+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8080102bd0 con 0x7f8080102180 2024-04-23T18:32:57.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.322+0000 7f806f7fe640 1 -- 172.21.15.32:0/1176786594 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f807009c640 con 0x7f8080102180 2024-04-23T18:32:57.315 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 msgr2=0x7f80680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 0x7f80680a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f80740058c0 tx=0x7f8074005850 comp rx=0 tx=0).stop 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 msgr2=0x7f80801a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f807000b4a0 tx=0x7f807000b960 comp rx=0 tx=0).stop 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 shutdown_connections 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f808019a7c0 0x7f808019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080107250 0x7f80801a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f80680a3d20 0x7f80680a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 --2- 172.21.15.32:0/1176786594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8080102180 0x7f80801a0650 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 >> 172.21.15.32:0/1176786594 conn(0x7f80800fda00 msgr2=0x7f80800ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 shutdown_connections 2024-04-23T18:32:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.324+0000 7f80848cb640 1 -- 172.21.15.32:0/1176786594 wait complete. 2024-04-23T18:32:57.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 -- 172.21.15.32:0/1946106417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff114101a50 msgr2=0x7ff114105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 --2- 172.21.15.32:0/1946106417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff114101a50 0x7ff114105a90 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff104009900 tx=0x7ff10402a410 comp rx=0 tx=0).stop 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 -- 172.21.15.32:0/1946106417 shutdown_connections 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 --2- 172.21.15.32:0/1946106417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 0x7ff11410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 --2- 172.21.15.32:0/1946106417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff114101a50 0x7ff114105a90 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 --2- 172.21.15.32:0/1946106417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1141010a0 0x7ff114101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 -- 172.21.15.32:0/1946106417 >> 172.21.15.32:0/1946106417 conn(0x7ff114077f40 msgr2=0x7ff114078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 -- 172.21.15.32:0/1946106417 shutdown_connections 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.440+0000 7ff11bae1640 1 -- 172.21.15.32:0/1946106417 wait complete. 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 Processor -- start 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 -- start start 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 0x7ff114198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff11419cf50 0x7ff11419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1141121f0 con 0x7ff1141010a0 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff114112070 con 0x7ff11419cf50 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11bae1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff114112370 con 0x7ff114106170 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11987d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11987d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47768/0 (socket says 172.21.15.32:47768) 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11987d640 1 -- 172.21.15.32:0/3671069095 learned_addr learned my addr 172.21.15.32:0/3671069095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.441+0000 7ff11a07e640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff11419cf50 0x7ff11419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11907c640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 0x7ff114198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 -- 172.21.15.32:0/3671069095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 msgr2=0x7ff114198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 0x7ff114198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 -- 172.21.15.32:0/3671069095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff11419cf50 msgr2=0x7ff11419d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff11419cf50 0x7ff11419d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 -- 172.21.15.32:0/3671069095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11419da40 con 0x7ff1141010a0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11987d640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d950 tx=0x7ff11000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff1141010a0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff110009c30 con 0x7ff1141010a0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11419dd30 con 0x7ff1141010a0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11000f620 con 0x7ff1141010a0 2024-04-23T18:32:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.442+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1141a5540 con 0x7ff1141010a0 2024-04-23T18:32:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.443+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff114102570 con 0x7ff1141010a0 2024-04-23T18:32:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.444+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff110010460 con 0x7ff1141010a0 2024-04-23T18:32:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.445+0000 7ff10affd640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 0x7ff1000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.445+0000 7ff11907c640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 0x7ff1000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.445+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7ff1100d03a0 con 0x7ff1141010a0 2024-04-23T18:32:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.446+0000 7ff11907c640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 0x7ff1000a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c00 tx=0x7ff104032ca0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.450+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff11009ca60 con 0x7ff1141010a0 2024-04-23T18:32:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.638+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1141999e0 con 0x7ff1141010a0 2024-04-23T18:32:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.639+0000 7ff10affd640 1 -- 172.21.15.32:0/3671069095 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7ff110009de0 con 0x7ff1141010a0 2024-04-23T18:32:57.631 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 msgr2=0x7ff1000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 0x7ff1000a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c00 tx=0x7ff104032ca0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 msgr2=0x7ff114198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d950 tx=0x7ff11000de10 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 shutdown_connections 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff11419cf50 0x7ff11419d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff114106170 0x7ff114198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7ff1000a3da0 0x7ff1000a6260 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 --2- 172.21.15.32:0/3671069095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1141010a0 0x7ff114198690 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 >> 172.21.15.32:0/3671069095 conn(0x7ff114077f40 msgr2=0x7ff1140fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 shutdown_connections 2024-04-23T18:32:57.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.641+0000 7ff11bae1640 1 -- 172.21.15.32:0/3671069095 wait complete. 2024-04-23T18:32:57.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 -- 172.21.15.32:0/3208623899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 msgr2=0x7f1bb4110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/3208623899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb4110ce0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009ca0 tx=0x7f1bac02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 -- 172.21.15.32:0/3208623899 shutdown_connections 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/3208623899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb4110ce0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/3208623899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4101780 0x7f1bb410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/3208623899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4100dd0 0x7f1bb41011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 -- 172.21.15.32:0/3208623899 >> 172.21.15.32:0/3208623899 conn(0x7f1bb4077f70 msgr2=0x7f1bb4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 -- 172.21.15.32:0/3208623899 shutdown_connections 2024-04-23T18:32:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.759+0000 7f1bb89d8640 1 -- 172.21.15.32:0/3208623899 wait complete. 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 Processor -- start 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 -- start start 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4100dd0 0x7f1bb41a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4101780 0x7f1bb41a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114500 con 0x7f1bb410e8f0 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114380 con 0x7f1bb4100dd0 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb89d8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114680 con 0x7f1bb4101780 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47790/0 (socket says 172.21.15.32:47790) 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.760+0000 7f1bb2d9d640 1 -- 172.21.15.32:0/1588628489 learned_addr learned my addr 172.21.15.32:0/1588628489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb259c640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4100dd0 0x7f1bb41a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb1d9b640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4101780 0x7f1bb41a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 -- 172.21.15.32:0/1588628489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4101780 msgr2=0x7f1bb41a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4101780 0x7f1bb41a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 -- 172.21.15.32:0/1588628489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4100dd0 msgr2=0x7f1bb41a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4100dd0 0x7f1bb41a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 -- 172.21.15.32:0/1588628489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb419b010 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb2d9d640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1bac005e30 tx=0x7f1bac0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac003bf0 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb419b2a0 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac003d90 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02fbd0 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.761+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb41a7520 con 0x7f1bb410e8f0 2024-04-23T18:32:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.762+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb406ae40 con 0x7f1bb410e8f0 2024-04-23T18:32:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.763+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1bac02fd70 con 0x7f1bb410e8f0 2024-04-23T18:32:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.764+0000 7f1ba37fe640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 0x7f1b9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.764+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f1bac0f1b40 con 0x7f1bb410e8f0 2024-04-23T18:32:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.764+0000 7f1bb259c640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 0x7f1b9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.764+0000 7f1bb259c640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 0x7f1b9c0a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4005fd0 tx=0x7f1ba400b390 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.769+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bac0be340 con 0x7f1bb410e8f0 2024-04-23T18:32:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.957+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1bb4109a80 con 0x7f1bb410e8f0 2024-04-23T18:32:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.957+0000 7f1ba37fe640 1 -- 172.21.15.32:0/1588628489 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f1bac0bdc40 con 0x7f1bb410e8f0 2024-04-23T18:32:57.950 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 msgr2=0x7f1b9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 0x7f1b9c0a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4005fd0 tx=0x7f1ba400b390 comp rx=0 tx=0).stop 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 msgr2=0x7f1bb419a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1bac005e30 tx=0x7f1bac0319f0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 shutdown_connections 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f1b9c0a3d20 0x7f1b9c0a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bb410e8f0 0x7f1bb419a840 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bb4101780 0x7f1bb41a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 --2- 172.21.15.32:0/1588628489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bb4100dd0 0x7f1bb41a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 >> 172.21.15.32:0/1588628489 conn(0x7f1bb4077f70 msgr2=0x7f1bb410fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 shutdown_connections 2024-04-23T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:57.959+0000 7f1bb89d8640 1 -- 172.21.15.32:0/1588628489 wait complete. 2024-04-23T18:32:57.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/525863689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:57.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1176786594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:57.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:57 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/3671069095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:57.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T18:32:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.073+0000 7fa65a835640 1 -- 172.21.15.32:0/1686383157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 msgr2=0x7fa654101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.073+0000 7fa65a835640 1 --2- 172.21.15.32:0/1686383157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa654101390 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa648009930 tx=0x7fa64802a420 comp rx=0 tx=0).stop 2024-04-23T18:32:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 -- 172.21.15.32:0/1686383157 shutdown_connections 2024-04-23T18:32:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- 172.21.15.32:0/1686383157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6541018d0 0x7fa65410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- 172.21.15.32:0/1686383157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa654101390 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- 172.21.15.32:0/1686383157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa654106f50 0x7fa654107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 -- 172.21.15.32:0/1686383157 >> 172.21.15.32:0/1686383157 conn(0x7fa654076f20 msgr2=0x7fa654077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 -- 172.21.15.32:0/1686383157 shutdown_connections 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 -- 172.21.15.32:0/1686383157 wait complete. 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 Processor -- start 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 -- start start 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6541018d0 0x7fa654198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.074+0000 7fa65a835640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa654106f50 0x7fa65419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa65a835640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa654112280 con 0x7fa654100f30 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa65a835640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa654112100 con 0x7fa6541018d0 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa65a835640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa654112400 con 0x7fa654106f50 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47812/0 (socket says 172.21.15.32:47812) 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 -- 172.21.15.32:0/2529846683 learned_addr learned my addr 172.21.15.32:0/2529846683 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa6537fe640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6541018d0 0x7fa654198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa658dd2640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa654106f50 0x7fa65419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 -- 172.21.15.32:0/2529846683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa654106f50 msgr2=0x7fa65419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa654106f50 0x7fa65419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 -- 172.21.15.32:0/2529846683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6541018d0 msgr2=0x7fa654198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6541018d0 0x7fa654198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 -- 172.21.15.32:0/2529846683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65419d690 con 0x7fa654100f30 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa653fff640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c920 tx=0x7fa64400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644007c70 con 0x7fa654100f30 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa644007e10 con 0x7fa654100f30 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65419d980 con 0x7fa654100f30 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644018820 con 0x7fa654100f30 2024-04-23T18:32:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.075+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541a5210 con 0x7fa654100f30 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.077+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa654068e00 con 0x7fa654100f30 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.078+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa64400f450 con 0x7fa654100f30 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.078+0000 7fa6517fa640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 0x7fa63c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.078+0000 7fa6537fe640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 0x7fa63c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.078+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fa6440d0860 con 0x7fa654100f30 2024-04-23T18:32:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.079+0000 7fa6537fe640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 0x7fa63c0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa648009900 tx=0x7fa648000bc0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.083+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa644014030 con 0x7fa654100f30 2024-04-23T18:32:58.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/525863689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:58.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1176786594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:58.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:57 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/3671069095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:58.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/525863689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:32:58.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1176786594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:32:58.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:57 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/3671069095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:32:58.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.282+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa6541999f0 con 0x7fa654100f30 2024-04-23T18:32:58.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.282+0000 7fa6517fa640 1 -- 172.21.15.32:0/2529846683 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v113) ==== 81+0+944 (secure 0 0 0) 0x7fa64409cf20 con 0x7fa654100f30 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/993405612 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3162050466 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3652066649 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3652066649 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1952921207 2024-04-24T18:05:55.984321+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1504393466 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3732407927 2024-04-24T18:05:55.984321+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/4149207751 2024-04-24T18:15:07.361281+0000 2024-04-23T18:32:58.275 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/2689415742 2024-04-24T18:05:55.984321+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2434896390 2024-04-24T18:05:55.984321+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/776389606 2024-04-24T18:05:28.817677+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/2689415742 2024-04-24T18:05:55.984321+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1265262618 2024-04-24T18:05:28.817677+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3094904085 2024-04-24T18:05:28.817677+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3656601632 2024-04-24T18:05:28.817677+0000 2024-04-23T18:32:58.276 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3656601632 2024-04-24T18:05:28.817677+0000 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:listed 16 entries 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 msgr2=0x7fa63c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 0x7fa63c0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa648009900 tx=0x7fa648000bc0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 msgr2=0x7fa6541986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c920 tx=0x7fa64400cde0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 shutdown_connections 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa654106f50 0x7fa65419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6541018d0 0x7fa654198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fa63c0a3d20 0x7fa63c0a61e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 --2- 172.21.15.32:0/2529846683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa654100f30 0x7fa6541986a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 >> 172.21.15.32:0/2529846683 conn(0x7fa654076f20 msgr2=0x7fa6540ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 shutdown_connections 2024-04-23T18:32:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:58.284+0000 7fa65a835640 1 -- 172.21.15.32:0/2529846683 wait complete. 2024-04-23T18:32:58.288 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T18:32:58.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /tmp/cephfs 2024-04-23T18:32:58.357 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:32:58.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.358 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-23T18:32:58.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.417 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-23T18:32:58.462 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-23T18:32:58.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-23T18:32:58.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-23T18:32:58.646 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T18:32:58.647 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-23T18:32:58.647 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:32:58.691 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19766 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/ef86b458f60718491b8f6a86d15a1cc5e9aa966e005677657c59407db740a75d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/ef86b458f60718491b8f6a86d15a1cc5e9aa966e005677657c59407db740a75d/diff,workdir=/var/lib/containers/storage/overlay/ef86b458f60718491b8f6a86d15a1cc5e9aa966e005677657c59407db740a75d/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bf5e135b897b322ec79bcc31259a59e200a3e6636d1eb4beb8cbdd41d3db59f9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/bf5e135b897b322ec79bcc31259a59e200a3e6636d1eb4beb8cbdd41d3db59f9/diff,workdir=/var/lib/containers/storage/overlay/bf5e135b897b322ec79bcc31259a59e200a3e6636d1eb4beb8cbdd41d3db59f9/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/e27656ac3a5fa38c02ac2696f019eb7463c619965c111af3b7b17683bd082882/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/e27656ac3a5fa38c02ac2696f019eb7463c619965c111af3b7b17683bd082882/diff,workdir=/var/lib/containers/storage/overlay/e27656ac3a5fa38c02ac2696f019eb7463c619965c111af3b7b17683bd082882/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/26dad30194edf4fee2a0e170259bce50308137ae6cf078d6847c2ca86480b358/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/26dad30194edf4fee2a0e170259bce50308137ae6cf078d6847c2ca86480b358/diff,workdir=/var/lib/containers/storage/overlay/26dad30194edf4fee2a0e170259bce50308137ae6cf078d6847c2ca86480b358/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f24cdefef6855738716dbab0920fbab9b81397396593b0ca264892e02e695f5a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/f24cdefef6855738716dbab0920fbab9b81397396593b0ca264892e02e695f5a/diff,workdir=/var/lib/containers/storage/overlay/f24cdefef6855738716dbab0920fbab9b81397396593b0ca264892e02e695f5a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0e4425c9ed00a32a0ccf56a0e70088727b9516fe6c7d532fd87a3ff02b56d6b0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/0e4425c9ed00a32a0ccf56a0e70088727b9516fe6c7d532fd87a3ff02b56d6b0/diff,workdir=/var/lib/containers/storage/overlay/0e4425c9ed00a32a0ccf56a0e70088727b9516fe6c7d532fd87a3ff02b56d6b0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/300b487e60ecec372e34b728cb4a56049bb617b5ee65d454096df7702d3b8fad/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/300b487e60ecec372e34b728cb4a56049bb617b5ee65d454096df7702d3b8fad/diff,workdir=/var/lib/containers/storage/overlay/300b487e60ecec372e34b728cb4a56049bb617b5ee65d454096df7702d3b8fad/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/74f99ce9f239d33c1bfc3c3d81cb823afab555389f60698e248122d4e1a355b3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/74f99ce9f239d33c1bfc3c3d81cb823afab555389f60698e248122d4e1a355b3/diff,workdir=/var/lib/containers/storage/overlay/74f99ce9f239d33c1bfc3c3d81cb823afab555389f60698e248122d4e1a355b3/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/70c2dc674db956cc2def4297332da0e51d2e19767df099ba26f1a03e6eb9522c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/70c2dc674db956cc2def4297332da0e51d2e19767df099ba26f1a03e6eb9522c/diff,workdir=/var/lib/containers/storage/overlay/70c2dc674db956cc2def4297332da0e51d2e19767df099ba26f1a03e6eb9522c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f848fe7472d3884197fc0a4bebf9df2361bff4b90c0d030abea139f03453de9d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OPBKRKSMDYNWT5PWVZ4UJ365UX:/var/lib/containers/storage/overlay/l/ZDJIGDPWLW25TNHEITCSJ4HUHJ,upperdir=/var/lib/containers/storage/overlay/f848fe7472d3884197fc0a4bebf9df2361bff4b90c0d030abea139f03453de9d/diff,workdir=/var/lib/containers/storage/overlay/f848fe7472d3884197fc0a4bebf9df2361bff4b90c0d030abea139f03453de9d/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-23T18:32:58.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.691 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-23T18:32:58.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:32:58.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-23T18:32:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-04-23T18:32:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:32:58.863 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T18:32:58.875 INFO:tasks.cephadm:Teardown begin 2024-04-23T18:32:58.875 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T18:32:58.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:32:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.055+0000 7f57cbbf7640 1 -- 172.21.15.32:0/1842302076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c4102190 msgr2=0x7f57c4102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.055+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/1842302076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c4102190 0x7f57c4102570 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009900 tx=0x7f57c002a400 comp rx=0 tx=0).stop 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 -- 172.21.15.32:0/1842302076 shutdown_connections 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/1842302076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/1842302076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57c4102b40 0x7f57c4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/1842302076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c4102190 0x7f57c4102570 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 -- 172.21.15.32:0/1842302076 >> 172.21.15.32:0/1842302076 conn(0x7f57c40fda30 msgr2=0x7f57c40ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 -- 172.21.15.32:0/1842302076 shutdown_connections 2024-04-23T18:32:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.056+0000 7f57cbbf7640 1 -- 172.21.15.32:0/1842302076 wait complete. 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 Processor -- start 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 -- start start 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57c4102190 0x7f57c410c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 0x7f57c41aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4075d70 con 0x7f57c410caa0 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4075bf0 con 0x7f57c4107260 2024-04-23T18:32:59.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.057+0000 7f57cbbf7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4075ef0 con 0x7f57c4102190 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57ca194640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 0x7f57c41aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57ca194640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 0x7f57c41aaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47832/0 (socket says 172.21.15.32:47832) 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57ca194640 1 -- 172.21.15.32:0/2540315657 learned_addr learned my addr 172.21.15.32:0/2540315657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 -- 172.21.15.32:0/2540315657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57c4102190 msgr2=0x7f57c410c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57c4102190 0x7f57c410c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 -- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 msgr2=0x7f57c41aaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 0x7f57c41aaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.058+0000 7f57c9192640 1 -- 172.21.15.32:0/2540315657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c41ab4d0 con 0x7f57c4107260 2024-04-23T18:32:59.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.060+0000 7f57c9192640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410c560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f57b400e9c0 tx=0x7f57b400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.061+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b400ccc0 con 0x7f57c4107260 2024-04-23T18:32:59.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.061+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c41ab760 con 0x7f57c4107260 2024-04-23T18:32:59.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.061+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c41abcd0 con 0x7f57c4107260 2024-04-23T18:32:59.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.061+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b40108e0 con 0x7f57c4107260 2024-04-23T18:32:59.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.061+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b40187a0 con 0x7f57c4107260 2024-04-23T18:32:59.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.062+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a8005290 con 0x7f57c4107260 2024-04-23T18:32:59.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:58 smithi064 ceph-mon[33062]: pgmap v562: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:59.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/1588628489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:59.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:58 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2529846683' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:32:59.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.064+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f57b4010430 con 0x7f57c4107260 2024-04-23T18:32:59.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.065+0000 7f57baffd640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 0x7f57b00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.066+0000 7f57c9993640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 0x7f57b00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.066+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f57b4099400 con 0x7f57c4107260 2024-04-23T18:32:59.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.066+0000 7f57c9993640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 0x7f57b00a62a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57c0005c20 tx=0x7f57c0031400 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.074+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57b409d490 con 0x7f57c4107260 2024-04-23T18:32:59.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:58 smithi040 ceph-mon[32626]: pgmap v562: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:59.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/1588628489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:59.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:58 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2529846683' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:32:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:58 smithi032 ceph-mon[29221]: pgmap v562: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:32:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/1588628489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:32:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:58 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2529846683' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:32:59.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.310+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57a8005580 con 0x7f57c4107260 2024-04-23T18:32:59.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.311+0000 7f57baffd640 1 -- 172.21.15.32:0/2540315657 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11857 (secure 0 0 0) 0x7f57b409cd90 con 0x7f57c4107260 2024-04-23T18:32:59.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:59.304 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:32:26.781873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:32:59.304 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 55 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 msgr2=0x7f57b00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 0x7f57b00a62a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57c0005c20 tx=0x7f57c0031400 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 msgr2=0x7f57c410c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410c560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f57b400e9c0 tx=0x7f57b400ee80 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 shutdown_connections 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f57b00a3de0 0x7f57b00a62a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c410caa0 0x7f57c41aaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c4107260 0x7f57c410c560 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 --2- 172.21.15.32:0/2540315657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57c4102190 0x7f57c410c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 >> 172.21.15.32:0/2540315657 conn(0x7f57c40fda30 msgr2=0x7f57c40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 shutdown_connections 2024-04-23T18:32:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.313+0000 7f57cbbf7640 1 -- 172.21.15.32:0/2540315657 wait complete. 2024-04-23T18:32:59.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- 172.21.15.32:0/1892710127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 msgr2=0x7f06b010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- 172.21.15.32:0/1892710127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b010dc70 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009900 tx=0x7f06a002a410 comp rx=0 tx=0).stop 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- 172.21.15.32:0/1892710127 shutdown_connections 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- 172.21.15.32:0/1892710127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b010dc70 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- 172.21.15.32:0/1892710127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 0x7f06b0101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- 172.21.15.32:0/1892710127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 0x7f06b01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- 172.21.15.32:0/1892710127 >> 172.21.15.32:0/1892710127 conn(0x7f06b0077c40 msgr2=0x7f06b0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- 172.21.15.32:0/1892710127 shutdown_connections 2024-04-23T18:32:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- 172.21.15.32:0/1892710127 wait complete. 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 Processor -- start 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- start start 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 0x7f06b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 0x7f06b019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0111090 con 0x7f06b0101a40 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0110f10 con 0x7f06b01010a0 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.426+0000 7f06b7212640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0111210 con 0x7f06b0106ed0 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47856/0 (socket says 172.21.15.32:47856) 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 -- 172.21.15.32:0/143497645 learned_addr learned my addr 172.21.15.32:0/143497645 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06b57af640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 0x7f06b019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06b4fae640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 0x7f06b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 -- 172.21.15.32:0/143497645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 msgr2=0x7f06b019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 0x7f06b019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 -- 172.21.15.32:0/143497645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 msgr2=0x7f06b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 0x7f06b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 -- 172.21.15.32:0/143497645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b019d6d0 con 0x7f06b0101a40 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06affff640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f06a400d950 tx=0x7f06a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4017030 con 0x7f06b0101a40 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06a4004020 con 0x7f06b0101a40 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b019d9c0 con 0x7f06b0101a40 2024-04-23T18:32:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a40047a0 con 0x7f06b0101a40 2024-04-23T18:32:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.427+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01a5250 con 0x7f06b0101a40 2024-04-23T18:32:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.428+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b0068e00 con 0x7f06b0101a40 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.429+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f06a4009a30 con 0x7f06b0101a40 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.430+0000 7f06adffb640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 0x7f06980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.430+0000 7f06b4fae640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 0x7f06980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.430+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f06a40d0230 con 0x7f06b0101a40 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.431+0000 7f06b4fae640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 0x7f06980a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06a800a300 tx=0x7f06a800a290 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.435+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06a409c890 con 0x7f06b0101a40 2024-04-23T18:32:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.665+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06b01a55b0 con 0x7f06b0101a40 2024-04-23T18:32:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.665+0000 7f06adffb640 1 -- 172.21.15.32:0/143497645 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11857 (secure 0 0 0) 0x7f06a409c190 con 0x7f06b0101a40 2024-04-23T18:32:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:59.658 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1839255789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1839255789},{"type":"v1","addr":"172.21.15.32:6835","nonce":1839255789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2912347592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2912347592},{"type":"v1","addr":"172.21.15.32:6837","nonce":2912347592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/156606909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":156606909},{"type":"v1","addr":"172.21.15.32:6839","nonce":156606909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/2029849926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":2029849926},{"type":"v1","addr":"172.21.15.32:6841","nonce":2029849926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/2951030033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2951030033},{"type":"v1","addr":"172.21.15.64:6833","nonce":2951030033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/768829407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":768829407},{"type":"v1","addr":"172.21.15.40:6833","nonce":768829407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:13:05.375853+0000","modified":"2024-04-23T18:32:26.781873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24551,"mds_1":24515,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/669117639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":669117639},{"type":"v1","addr":"172.21.15.64:6835","nonce":669117639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/1068324383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1068324383},{"type":"v1","addr":"172.21.15.40:6835","nonce":1068324383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6839/1242971440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1242971440},{"type":"v1","addr":"172.21.15.64:6839","nonce":1242971440}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1154751886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1154751886},{"type":"v1","addr":"172.21.15.40:6837","nonce":1154751886}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2479383965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2479383965},{"type":"v1","addr":"172.21.15.40:6839","nonce":2479383965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26674":{"gid":26674,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1665828723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1665828723},{"type":"v1","addr":"172.21.15.64:6837","nonce":1665828723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24551,"qdb_cluster":[24551,24499,24515]},"id":1}]} 2024-04-23T18:32:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 55 2024-04-23T18:32:59.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 msgr2=0x7f06980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 0x7f06980a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06a800a300 tx=0x7f06a800a290 comp rx=0 tx=0).stop 2024-04-23T18:32:59.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 msgr2=0x7f06b0198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f06a400d950 tx=0x7f06a400de10 comp rx=0 tx=0).stop 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 shutdown_connections 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06b0106ed0 0x7f06b019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f06980a3da0 0x7f06980a6260 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b0101a40 0x7f06b0198c20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 --2- 172.21.15.32:0/143497645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b01010a0 0x7f06b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 >> 172.21.15.32:0/143497645 conn(0x7f06b0077c40 msgr2=0x7f06b00693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 shutdown_connections 2024-04-23T18:32:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.668+0000 7f06b7212640 1 -- 172.21.15.32:0/143497645 wait complete. 2024-04-23T18:32:59.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 -- 172.21.15.32:0/1003931453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c106170 msgr2=0x7f8a1c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 --2- 172.21.15.32:0/1003931453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c106170 0x7f8a1c10dd40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009ca0 tx=0x7f8a1402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 -- 172.21.15.32:0/1003931453 shutdown_connections 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 --2- 172.21.15.32:0/1003931453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c106170 0x7f8a1c10dd40 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 --2- 172.21.15.32:0/1003931453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c101a50 0x7f8a1c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 --2- 172.21.15.32:0/1003931453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c1010a0 0x7f8a1c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 -- 172.21.15.32:0/1003931453 >> 172.21.15.32:0/1003931453 conn(0x7f8a1c077f50 msgr2=0x7f8a1c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 -- 172.21.15.32:0/1003931453 shutdown_connections 2024-04-23T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.781+0000 7f8a20d44640 1 -- 172.21.15.32:0/1003931453 wait complete. 2024-04-23T18:32:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 Processor -- start 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 -- start start 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c1010a0 0x7f8a1c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c106170 0x7f8a1c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c1110c0 con 0x7f8a1c19cf60 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c110f40 con 0x7f8a1c106170 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.782+0000 7f8a20d44640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c111240 con 0x7f8a1c1010a0 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47860/0 (socket says 172.21.15.32:47860) 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 -- 172.21.15.32:0/917659167 learned_addr learned my addr 172.21.15.32:0/917659167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1a59c640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c1010a0 0x7f8a1c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a11d9b640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c106170 0x7f8a1c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 -- 172.21.15.32:0/917659167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c1010a0 msgr2=0x7f8a1c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c1010a0 0x7f8a1c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 -- 172.21.15.32:0/917659167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c106170 msgr2=0x7f8a1c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c106170 0x7f8a1c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 -- 172.21.15.32:0/917659167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1c19da50 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a1ad9d640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8a140098d0 tx=0x7f8a140317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14003b70 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1c19dce0 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a14003d10 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1402fa20 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.783+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a1c1a5530 con 0x7f8a1c19cf60 2024-04-23T18:32:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.785+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a1c069250 con 0x7f8a1c19cf60 2024-04-23T18:32:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.785+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8a1402fc20 con 0x7f8a1c19cf60 2024-04-23T18:32:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.786+0000 7f8a13fff640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 0x7f8a080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.786+0000 7f8a1a59c640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 0x7f8a080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.786+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f8a140f1c10 con 0x7f8a1c19cf60 2024-04-23T18:32:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.787+0000 7f8a1a59c640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 0x7f8a080a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c009bd0 tx=0x7f8a0c005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:59.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.791+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a140be380 con 0x7f8a1c19cf60 2024-04-23T18:32:59.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.977+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a1c101a50 con 0x7f8a1c19cf60 2024-04-23T18:32:59.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.978+0000 7f8a13fff640 1 -- 172.21.15.32:0/917659167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23004 (secure 0 0 0) 0x7f8a140bdc80 con 0x7f8a1c19cf60 2024-04-23T18:32:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-23T18:32:59.971 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"fsid":"cdb712f8-019b-11ef-bc93-c7b262605968","created":"2024-04-23T18:04:51.553089+0000","modified":"2024-04-23T18:32:10.775144+0000","last_up_change":"2024-04-23T18:12:14.796951+0000","last_in_change":"2024-04-23T18:11:54.105745+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:08:22.076400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:13:03.563177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:13:04.331221+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a71e039-95f2-441c-86d2-65a3c98560e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6803","nonce":1995005420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6805","nonce":1995005420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6809","nonce":1995005420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1995005420},{"type":"v1","addr":"172.21.15.32:6807","nonce":1995005420}]},"public_addr":"172.21.15.32:6803/1995005420","cluster_addr":"172.21.15.32:6805/1995005420","heartbeat_back_addr":"172.21.15.32:6809/1995005420","heartbeat_front_addr":"172.21.15.32:6807/1995005420","state":["exists","up"]},{"osd":1,"uuid":"780165d2-56a4-4fee-98a0-bc98ba8c9ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6801","nonce":2541914444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6803","nonce":2541914444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6807","nonce":2541914444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2541914444},{"type":"v1","addr":"172.21.15.40:6805","nonce":2541914444}]},"public_addr":"172.21.15.40:6801/2541914444","cluster_addr":"172.21.15.40:6803/2541914444","heartbeat_back_addr":"172.21.15.40:6807/2541914444","heartbeat_front_addr":"172.21.15.40:6805/2541914444","state":["exists","up"]},{"osd":2,"uuid":"50d71493-93bf-4dfe-ac4b-6a378dd2e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6801","nonce":3950848968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6803","nonce":3950848968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6807","nonce":3950848968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3950848968},{"type":"v1","addr":"172.21.15.64:6805","nonce":3950848968}]},"public_addr":"172.21.15.64:6801/3950848968","cluster_addr":"172.21.15.64:6803/3950848968","heartbeat_back_addr":"172.21.15.64:6807/3950848968","heartbeat_front_addr":"172.21.15.64:6805/3950848968","state":["exists","up"]},{"osd":3,"uuid":"253ddac0-f39a-459c-bef1-2d3bd28a2b2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6811","nonce":2339361307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6813","nonce":2339361307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6817","nonce":2339361307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2339361307},{"type":"v1","addr":"172.21.15.32:6815","nonce":2339361307}]},"public_addr":"172.21.15.32:6811/2339361307","cluster_addr":"172.21.15.32:6813/2339361307","heartbeat_back_addr":"172.21.15.32:6817/2339361307","heartbeat_front_addr":"172.21.15.32:6815/2339361307","state":["exists","up"]},{"osd":4,"uuid":"c0f898fa-45b8-4ebd-bf04-b5758d43461e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6809","nonce":3289252322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6811","nonce":3289252322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6815","nonce":3289252322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3289252322},{"type":"v1","addr":"172.21.15.40:6813","nonce":3289252322}]},"public_addr":"172.21.15.40:6809/3289252322","cluster_addr":"172.21.15.40:6811/3289252322","heartbeat_back_addr":"172.21.15.40:6815/3289252322","heartbeat_front_addr":"172.21.15.40:6813/3289252322","state":["exists","up"]},{"osd":5,"uuid":"9de82db1-b3dc-4181-9d27-fa440325720c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6809","nonce":1980421875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6811","nonce":1980421875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6815","nonce":1980421875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1980421875},{"type":"v1","addr":"172.21.15.64:6813","nonce":1980421875}]},"public_addr":"172.21.15.64:6809/1980421875","cluster_addr":"172.21.15.64:6811/1980421875","heartbeat_back_addr":"172.21.15.64:6815/1980421875","heartbeat_front_addr":"172.21.15.64:6813/1980421875","state":["exists","up"]},{"osd":6,"uuid":"ea694844-38fa-4af5-842f-1d69ba8d0679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6819","nonce":3191318575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6821","nonce":3191318575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6825","nonce":3191318575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3191318575},{"type":"v1","addr":"172.21.15.32:6823","nonce":3191318575}]},"public_addr":"172.21.15.32:6819/3191318575","cluster_addr":"172.21.15.32:6821/3191318575","heartbeat_back_addr":"172.21.15.32:6825/3191318575","heartbeat_front_addr":"172.21.15.32:6823/3191318575","state":["exists","up"]},{"osd":7,"uuid":"c6eaac83-e79b-4a1e-8ab6-31b047aca03b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6817","nonce":142928766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6819","nonce":142928766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6823","nonce":142928766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":142928766},{"type":"v1","addr":"172.21.15.40:6821","nonce":142928766}]},"public_addr":"172.21.15.40:6817/142928766","cluster_addr":"172.21.15.40:6819/142928766","heartbeat_back_addr":"172.21.15.40:6823/142928766","heartbeat_front_addr":"172.21.15.40:6821/142928766","state":["exists","up"]},{"osd":8,"uuid":"bb2a922c-0367-4bc6-8556-983cf6b11b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6817","nonce":3639114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6819","nonce":3639114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6823","nonce":3639114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3639114663},{"type":"v1","addr":"172.21.15.64:6821","nonce":3639114663}]},"public_addr":"172.21.15.64:6817/3639114663","cluster_addr":"172.21.15.64:6819/3639114663","heartbeat_back_addr":"172.21.15.64:6823/3639114663","heartbeat_front_addr":"172.21.15.64:6821/3639114663","state":["exists","up"]},{"osd":9,"uuid":"2db0b880-e61e-46ca-9826-421246e440fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6827","nonce":3112793650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6829","nonce":3112793650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6833","nonce":3112793650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3112793650},{"type":"v1","addr":"172.21.15.32:6831","nonce":3112793650}]},"public_addr":"172.21.15.32:6827/3112793650","cluster_addr":"172.21.15.32:6829/3112793650","heartbeat_back_addr":"172.21.15.32:6833/3112793650","heartbeat_front_addr":"172.21.15.32:6831/3112793650","state":["exists","up"]},{"osd":10,"uuid":"b2116ca5-2fe8-405c-bfeb-0eac6b746fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6825","nonce":4006751902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6827","nonce":4006751902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6831","nonce":4006751902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4006751902},{"type":"v1","addr":"172.21.15.40:6829","nonce":4006751902}]},"public_addr":"172.21.15.40:6825/4006751902","cluster_addr":"172.21.15.40:6827/4006751902","heartbeat_back_addr":"172.21.15.40:6831/4006751902","heartbeat_front_addr":"172.21.15.40:6829/4006751902","state":["exists","up"]},{"osd":11,"uuid":"4d5a0032-0c95-4c63-b06a-340f80f978de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6825","nonce":4083016219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6827","nonce":4083016219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6831","nonce":4083016219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":4083016219},{"type":"v1","addr":"172.21.15.64:6829","nonce":4083016219}]},"public_addr":"172.21.15.64:6825/4083016219","cluster_addr":"172.21.15.64:6827/4083016219","heartbeat_back_addr":"172.21.15.64:6831/4083016219","heartbeat_front_addr":"172.21.15.64:6829/4083016219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:31.953040+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:07:55.418486+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:19.876040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:08:44.219778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:09.197310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:34.900902+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:00.395704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:26.094428+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:52.220482+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:19.360416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:45.660937+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:13.136521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/993405612":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3162050466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6800/3652066649":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/1952921207":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1504393466":"2024-04-24T18:15:07.361281+0000","172.21.15.32:0/3732407927":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/4149207751":"2024-04-24T18:15:07.361281+0000","172.21.15.32:6801/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/2434896390":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/776389606":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/2689415742":"2024-04-24T18:05:55.984321+0000","172.21.15.32:0/1265262618":"2024-04-24T18:05:28.817677+0000","172.21.15.32:0/3094904085":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6800/3656601632":"2024-04-24T18:05:28.817677+0000","172.21.15.32:6801/3656601632":"2024-04-24T18:05:28.817677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 msgr2=0x7f8a080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 0x7f8a080a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c009bd0 tx=0x7f8a0c005f90 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 msgr2=0x7f8a1c19d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8a140098d0 tx=0x7f8a140317b0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 shutdown_connections 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f8a080a3d20 0x7f8a080a61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a1c19cf60 0x7f8a1c19d410 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a1c106170 0x7f8a1c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 --2- 172.21.15.32:0/917659167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a1c1010a0 0x7f8a1c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 >> 172.21.15.32:0/917659167 conn(0x7f8a1c077f50 msgr2=0x7f8a1c06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:59.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.980+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 shutdown_connections 2024-04-23T18:32:59.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:32:59.981+0000 7f8a20d44640 1 -- 172.21.15.32:0/917659167 wait complete. 2024-04-23T18:32:59.982 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:59 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2540315657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:59.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:32:59 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/143497645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:59.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T18:33:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 -- 172.21.15.32:0/2618180201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c107330 msgr2=0x7faa8c10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 --2- 172.21.15.32:0/2618180201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c107330 0x7faa8c10ef00 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7faa84009ca0 tx=0x7faa8402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 -- 172.21.15.32:0/2618180201 shutdown_connections 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 --2- 172.21.15.32:0/2618180201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c107330 0x7faa8c10ef00 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 --2- 172.21.15.32:0/2618180201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c102c10 0x7faa8c106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 --2- 172.21.15.32:0/2618180201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c102260 0x7faa8c102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 -- 172.21.15.32:0/2618180201 >> 172.21.15.32:0/2618180201 conn(0x7faa8c0fda80 msgr2=0x7faa8c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 -- 172.21.15.32:0/2618180201 shutdown_connections 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 -- 172.21.15.32:0/2618180201 wait complete. 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.096+0000 7faa9174a640 1 Processor -- start 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 -- start start 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c107330 0x7faa8c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c074cd0 0x7faa8c1a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c112290 con 0x7faa8c102260 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c112110 con 0x7faa8c107330 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa9174a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c112410 con 0x7faa8c074cd0 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47872/0 (socket says 172.21.15.32:47872) 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 -- 172.21.15.32:0/159871332 learned_addr learned my addr 172.21.15.32:0/159871332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa83fff640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c107330 0x7faa8c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8b7fe640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c074cd0 0x7faa8c1a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 -- 172.21.15.32:0/159871332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c074cd0 msgr2=0x7faa8c1a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c074cd0 0x7faa8c1a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 -- 172.21.15.32:0/159871332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c107330 msgr2=0x7faa8c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c107330 0x7faa8c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 -- 172.21.15.32:0/159871332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c1a7080 con 0x7faa8c102260 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa8affd640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00d770 tx=0x7faa7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c017070 con 0x7faa8c102260 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.097+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa7c0054d0 con 0x7faa8c102260 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.098+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c1a7370 con 0x7faa8c102260 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.098+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c004c30 con 0x7faa8c102260 2024-04-23T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.098+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8c1a78c0 con 0x7faa8c102260 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.099+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8c069250 con 0x7faa8c102260 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.100+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7faa7c004dd0 con 0x7faa8c102260 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.101+0000 7faa88ff9640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 0x7faa780a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.101+0000 7faa83fff640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 0x7faa780a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.101+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7faa7c0d0220 con 0x7faa8c102260 2024-04-23T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.101+0000 7faa83fff640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 0x7faa780a6290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faa8c075ce0 tx=0x7faa7400a2d0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.105+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa7c09c8e0 con 0x7faa8c102260 2024-04-23T18:33:00.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:59 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2540315657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:00.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:32:59 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/143497645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:00.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:59 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2540315657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:00.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:32:59 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/143497645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:00.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.331+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faa8c10a560 con 0x7faa8c102260 2024-04-23T18:33:00.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.333+0000 7faa88ff9640 1 -- 172.21.15.32:0/159871332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) ==== 161+0+0 (secure 0 0 0) 0x7faa7c09c1e0 con 0x7faa8c102260 2024-04-23T18:33:00.326 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T18:33:00.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 msgr2=0x7faa780a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 0x7faa780a6290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faa8c075ce0 tx=0x7faa7400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 msgr2=0x7faa8c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00d770 tx=0x7faa7c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 shutdown_connections 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa8c074cd0 0x7faa8c1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8c107330 0x7faa8c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faa780a3dd0 0x7faa780a6290 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 --2- 172.21.15.32:0/159871332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa8c102260 0x7faa8c074250 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 >> 172.21.15.32:0/159871332 conn(0x7faa8c0fda80 msgr2=0x7faa8c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 shutdown_connections 2024-04-23T18:33:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.336+0000 7faa9174a640 1 -- 172.21.15.32:0/159871332 wait complete. 2024-04-23T18:33:00.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 -- 172.21.15.32:0/2500685072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae8106170 msgr2=0x7f6ae810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 --2- 172.21.15.32:0/2500685072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae8106170 0x7f6ae810dd40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009ca0 tx=0x7f6ae002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 -- 172.21.15.32:0/2500685072 shutdown_connections 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 --2- 172.21.15.32:0/2500685072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae8106170 0x7f6ae810dd40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 --2- 172.21.15.32:0/2500685072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101a50 0x7f6ae8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 --2- 172.21.15.32:0/2500685072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 0x7f6ae8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 -- 172.21.15.32:0/2500685072 >> 172.21.15.32:0/2500685072 conn(0x7f6ae8077f40 msgr2=0x7f6ae8078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.453+0000 7f6aee893640 1 -- 172.21.15.32:0/2500685072 shutdown_connections 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 -- 172.21.15.32:0/2500685072 wait complete. 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 Processor -- start 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 -- start start 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 0x7f6ae81995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8106170 0x7f6ae8199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81110e0 con 0x7f6ae819a0b0 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8110f60 con 0x7f6ae8106170 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.454+0000 7f6aee893640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8111260 con 0x7f6ae81010a0 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47888/0 (socket says 172.21.15.32:47888) 2024-04-23T18:33:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 0x7f6ae81995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 -- 172.21.15.32:0/6953405 learned_addr learned my addr 172.21.15.32:0/6953405 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6ae77fe640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8106170 0x7f6ae8199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 -- 172.21.15.32:0/6953405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 msgr2=0x7f6ae81995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 0x7f6ae81995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 -- 172.21.15.32:0/6953405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8106170 msgr2=0x7f6ae8199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8106170 0x7f6ae8199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 -- 172.21.15.32:0/6953405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae81a4e60 con 0x7f6ae819a0b0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aece30640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00098d0 tx=0x7f6ae00317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0003b70 con 0x7f6ae819a0b0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae81a50f0 con 0x7f6ae819a0b0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae0003d10 con 0x7f6ae819a0b0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81a5640 con 0x7f6ae819a0b0 2024-04-23T18:33:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.455+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae002fa20 con 0x7f6ae819a0b0 2024-04-23T18:33:00.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.456+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae806ac10 con 0x7f6ae819a0b0 2024-04-23T18:33:00.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.458+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6ae002fbc0 con 0x7f6ae819a0b0 2024-04-23T18:33:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.459+0000 7f6ae57fa640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 0x7f6ad40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.459+0000 7f6ae7fff640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 0x7f6ad40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.459+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8698+0+0 (secure 0 0 0) 0x7f6ae00f1f20 con 0x7f6ae819a0b0 2024-04-23T18:33:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.459+0000 7f6ae7fff640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 0x7f6ad40a61e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009bd0 tx=0x7f6ad8005f90 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:00.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.462+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ae00be5c0 con 0x7f6ae819a0b0 2024-04-23T18:33:00.670 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mds[63936]: -- 172.21.15.64:0/561383526 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a65edafa40 con 0x55a65ed83000 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: starting mds.f at 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mds[70432]: -- 172.21.15.64:0/61563890 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a58d4e7a40 con 0x55a58d4bac00 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: starting mds.c at 2024-04-23T18:33:00.671 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.672 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.672 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mds[64557]: -- 172.21.15.64:0/182359407 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556259759a40 con 0x55625972c800 2024-04-23T18:33:00.672 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: starting mds.i at 2024-04-23T18:33:00.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:00.691+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6ae81093a0 con 0x7f6ae819a0b0 2024-04-23T18:33:00.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a[29217]: 2024-04-23T18:33:00.332+0000 7f59bd440700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:33:00.802 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.802 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.802 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mds[62814]: -- 172.21.15.40:0/1331847626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642d586ba40 con 0x5642d583ec00 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: starting mds.b at 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mds[62508]: -- 172.21.15.40:0/3467372996 <== mon.1 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56479326da40 con 0x564793240800 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: starting mds.h at 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e[62197]: ignoring --setuser ceph since I am not root 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e[62197]: ignoring --setgroup ceph since I am not root 2024-04-23T18:33:00.803 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mds[62201]: -- 172.21.15.40:0/1481895600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0d6b73a40 con 0x55f0d6b47000 2024-04-23T18:33:00.804 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e[62197]: starting mds.e at 2024-04-23T18:33:01.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: pgmap v563: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-23T18:33:01.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/917659167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:33:01.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:00 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:33:01.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: pgmap v563: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/917659167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:33:01.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:00 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: pgmap v563: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/917659167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:33:01.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:33:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/159871332' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:33:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:33:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:33:01.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:00 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:33:01.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.335+0000 7f6ae57fa640 1 -- 172.21.15.32:0/6953405 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) ==== 104+0+0 (secure 0 0 0) 0x7f6ae00bdec0 con 0x7f6ae819a0b0 2024-04-23T18:33:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 msgr2=0x7f6ad40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 0x7f6ad40a61e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009bd0 tx=0x7f6ad8005f90 comp rx=0 tx=0).stop 2024-04-23T18:33:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 msgr2=0x7f6ae81a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00098d0 tx=0x7f6ae00317b0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 shutdown_connections 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7f6ad40a3d20 0x7f6ad40a61e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ae819a0b0 0x7f6ae81a4660 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8106170 0x7f6ae8199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 --2- 172.21.15.32:0/6953405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ae81010a0 0x7f6ae81995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 >> 172.21.15.32:0/6953405 conn(0x7f6ae8077f40 msgr2=0x7f6ae80ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 shutdown_connections 2024-04-23T18:33:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.338+0000 7f6aee893640 1 -- 172.21.15.32:0/6953405 wait complete. 2024-04-23T18:33:01.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 -- 172.21.15.32:0/896139666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde801044f0 msgr2=0x7fde8010eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 --2- 172.21.15.32:0/896139666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde801044f0 0x7fde8010eef0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fde78009cd0 tx=0x7fde7802a8e0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 -- 172.21.15.32:0/896139666 shutdown_connections 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 --2- 172.21.15.32:0/896139666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde801044f0 0x7fde8010eef0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 --2- 172.21.15.32:0/896139666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 --2- 172.21.15.32:0/896139666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde80102270 0x7fde80102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 -- 172.21.15.32:0/896139666 >> 172.21.15.32:0/896139666 conn(0x7fde800fda10 msgr2=0x7fde800ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 -- 172.21.15.32:0/896139666 shutdown_connections 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 -- 172.21.15.32:0/896139666 wait complete. 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.467+0000 7fde860d9640 1 Processor -- start 2024-04-23T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 -- start start 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 0x7fde8019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde80112270 con 0x7fde80102270 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde801120f0 con 0x7fde80103b50 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde860d9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fde801123f0 con 0x7fde801044f0 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47910/0 (socket says 172.21.15.32:47910) 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 -- 172.21.15.32:0/2656029846 learned_addr learned my addr 172.21.15.32:0/2656029846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7effd640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7ffff640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 0x7fde8019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 -- 172.21.15.32:0/2656029846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 msgr2=0x7fde8019d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 0x7fde8019d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 -- 172.21.15.32:0/2656029846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 msgr2=0x7fde80198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 -- 172.21.15.32:0/2656029846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8019d740 con 0x7fde80102270 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7effd640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7ffff640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 0x7fde8019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.468+0000 7fde7f7fe640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fde7000d950 tx=0x7fde7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.469+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70017070 con 0x7fde80102270 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.469+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde70009c30 con 0x7fde80102270 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.469+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8019da30 con 0x7fde80102270 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.469+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7000f620 con 0x7fde80102270 2024-04-23T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.469+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde801a52c0 con 0x7fde80102270 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.470+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde80199aa0 con 0x7fde80102270 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.471+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fde70010460 con 0x7fde80102270 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.472+0000 7fde7cff9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 0x7fde680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.472+0000 7fde7effd640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 0x7fde680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.472+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8698+0+0 (secure 0 0 0) 0x7fde700d05c0 con 0x7fde80102270 2024-04-23T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.472+0000 7fde7effd640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 0x7fde680a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde80100070 tx=0x7fde74005e50 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.476+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde7009cbb0 con 0x7fde80102270 2024-04-23T18:33:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:01.664+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fde8010a540 con 0x7fde80102270 2024-04-23T18:33:02.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:33:02.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:33:02.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: Cluster is now healthy 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6834/606897118,v1:172.21.15.64:6836/606897118] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6836/1124638054,v1:172.21.15.40:6838/1124638054] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6835/3835816904,v1:172.21.15.40:6839/3835816904] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.40:6834/529916442,v1:172.21.15.40:6837/529916442] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6835/2035596036,v1:172.21.15.64:6838/2035596036] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: mds.? [v2:172.21.15.64:6837/3066217827,v1:172.21.15.64:6839/3066217827] up:boot 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: fsmap 12 up:standby 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:33:02.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.117 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:01 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: Cluster is now healthy 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6834/606897118,v1:172.21.15.64:6836/606897118] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6836/1124638054,v1:172.21.15.40:6838/1124638054] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6835/3835816904,v1:172.21.15.40:6839/3835816904] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.40:6834/529916442,v1:172.21.15.40:6837/529916442] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6835/2035596036,v1:172.21.15.64:6838/2035596036] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: mds.? [v2:172.21.15.64:6837/3066217827,v1:172.21.15.64:6839/3066217827] up:boot 2024-04-23T18:33:02.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: fsmap 12 up:standby 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.172 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:01 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:02.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:33:02.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:33:02.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: Cluster is now healthy 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/6953405' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6834/606897118,v1:172.21.15.64:6836/606897118] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6836/1124638054,v1:172.21.15.40:6838/1124638054] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6835/3835816904,v1:172.21.15.40:6839/3835816904] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.40:6834/529916442,v1:172.21.15.40:6837/529916442] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6835/2035596036,v1:172.21.15.64:6838/2035596036] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: mds.? [v2:172.21.15.64:6837/3066217827,v1:172.21.15.64:6839/3066217827] up:boot 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: fsmap 12 up:standby 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:33:02.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='mgr.15294 172.21.15.32:0/1252946046' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:33:02.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:01 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.340+0000 7fde7cff9640 1 -- 172.21.15.32:0/2656029846 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7fde70009de0 con 0x7fde80102270 2024-04-23T18:33:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-04-23T18:33:02.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 msgr2=0x7fde680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:02.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 0x7fde680a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde80100070 tx=0x7fde74005e50 comp rx=0 tx=0).stop 2024-04-23T18:33:02.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 msgr2=0x7fde80198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fde7000d950 tx=0x7fde7000de10 comp rx=0 tx=0).stop 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 shutdown_connections 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde801044f0 0x7fde8019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde80103b50 0x7fde80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7fde680a3d20 0x7fde680a61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 --2- 172.21.15.32:0/2656029846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80102270 0x7fde80198750 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 >> 172.21.15.32:0/2656029846 conn(0x7fde800fda10 msgr2=0x7fde800fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 shutdown_connections 2024-04-23T18:33:02.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.345+0000 7fde860d9640 1 -- 172.21.15.32:0/2656029846 wait complete. 2024-04-23T18:33:02.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T18:33:02.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.474+0000 7fab1400b640 1 -- 172.21.15.32:0/750578249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 msgr2=0x7fab0c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:02.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.474+0000 7fab1400b640 1 --2- 172.21.15.32:0/750578249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c10dd30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7faafc009900 tx=0x7faafc02a410 comp rx=0 tx=0).stop 2024-04-23T18:33:02.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- 172.21.15.32:0/750578249 shutdown_connections 2024-04-23T18:33:02.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- 172.21.15.32:0/750578249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c10dd30 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- 172.21.15.32:0/750578249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c101a40 0x7fab0c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- 172.21.15.32:0/750578249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 0x7fab0c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- 172.21.15.32:0/750578249 >> 172.21.15.32:0/750578249 conn(0x7fab0c078000 msgr2=0x7fab0c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- 172.21.15.32:0/750578249 shutdown_connections 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- 172.21.15.32:0/750578249 wait complete. 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 Processor -- start 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- start start 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 0x7fab0c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c19cf80 0x7fab0c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c112220 con 0x7fab0c106160 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1120a0 con 0x7fab0c19cf80 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.475+0000 7fab1400b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1123a0 con 0x7fab0c101090 2024-04-23T18:33:02.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47920/0 (socket says 172.21.15.32:47920) 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 -- 172.21.15.32:0/240895392 learned_addr learned my addr 172.21.15.32:0/240895392 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab125a8640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c19cf80 0x7fab0c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab11da7640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 0x7fab0c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 -- 172.21.15.32:0/240895392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 msgr2=0x7fab0c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 0x7fab0c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 -- 172.21.15.32:0/240895392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c19cf80 msgr2=0x7fab0c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c19cf80 0x7fab0c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 -- 172.21.15.32:0/240895392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c19da70 con 0x7fab0c106160 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab115a6640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fab0000c970 tx=0x7fab0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00014070 con 0x7fab0c106160 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab00004440 con 0x7fab0c106160 2024-04-23T18:33:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c19dd60 con 0x7fab0c106160 2024-04-23T18:33:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00003e50 con 0x7fab0c106160 2024-04-23T18:33:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.476+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c1a5550 con 0x7fab0c106160 2024-04-23T18:33:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.477+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab0c068e00 con 0x7fab0c106160 2024-04-23T18:33:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.478+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fab00017070 con 0x7fab0c106160 2024-04-23T18:33:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.479+0000 7fab0affd640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 0x7faaf40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.479+0000 7fab11da7640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 0x7faaf40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.479+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8144+0+0 (secure 0 0 0) 0x7fab000d01c0 con 0x7fab0c106160 2024-04-23T18:33:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.480+0000 7fab11da7640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 0x7faaf40a6380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab04009f20 tx=0x7fab040094f0 comp rx=0 tx=0).ready entity=mgr.15294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:02.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.483+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab0001e030 con 0x7fab0c106160 2024-04-23T18:33:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:02.672+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fab0c101a40 con 0x7fab0c106160 2024-04-23T18:33:03.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:02 smithi040 ceph-mon[32626]: pgmap v565: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-04-23T18:33:03.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:33:03.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:02 smithi040 ceph-mon[32626]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:33:03.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:02 smithi040 ceph-mon[32626]: from='client.? 172.21.15.32:0/240895392' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:03.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:02 smithi064 ceph-mon[33062]: pgmap v565: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-04-23T18:33:03.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:33:03.170 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:02 smithi064 ceph-mon[33062]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:33:03.171 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:02 smithi064 ceph-mon[33062]: from='client.? 172.21.15.32:0/240895392' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:02 smithi032 ceph-mon[29221]: pgmap v565: 97 pgs: 97 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-04-23T18:33:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/2656029846' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:33:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:02 smithi032 ceph-mon[29221]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:33:03.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:02 smithi032 ceph-mon[29221]: from='client.? 172.21.15.32:0/240895392' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:33:03.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.342+0000 7fab0affd640 1 -- 172.21.15.32:0/240895392 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7fab0009ca30 con 0x7fab0c106160 2024-04-23T18:33:03.334 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 msgr2=0x7faaf40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 0x7faaf40a6380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab04009f20 tx=0x7fab040094f0 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 msgr2=0x7fab0c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fab0000c970 tx=0x7fab0000ce30 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 shutdown_connections 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab0c19cf80 0x7fab0c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:6800/3410706145,v1:172.21.15.32:6801/3410706145] conn(0x7faaf40a3ec0 0x7faaf40a6380 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c106160 0x7fab0c198c00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 --2- 172.21.15.32:0/240895392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab0c101090 0x7fab0c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 >> 172.21.15.32:0/240895392 conn(0x7fab0c078000 msgr2=0x7fab0c0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.346+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 shutdown_connections 2024-04-23T18:33:03.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-23T18:33:03.347+0000 7fab1400b640 1 -- 172.21.15.32:0/240895392 wait complete. 2024-04-23T18:33:03.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:33:03.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:33:03.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:33:03.462 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T18:33:03.462 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:33:03.480 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:33:03.497 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:33:03.519 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T18:33:03.519 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T18:33:03.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:33:03.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:03 smithi032 systemd[1]: Stopping Ceph mon.a for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:04.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a[29217]: 2024-04-23T18:33:04.022+0000 7f59c2c4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:04.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a[29217]: 2024-04-23T18:33:04.022+0000 7f59c2c4b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T18:33:04.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 podman[95928]: 2024-04-23 18:33:04.168207794 +0000 UTC m=+0.286624429 container died 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-23T18:33:04.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 podman[95928]: 2024-04-23 18:33:04.589225843 +0000 UTC m=+0.707642480 container cleanup 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:33:04.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 bash[95928]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:33:05.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 23 18:33:04 smithi032 podman[95938]: 2024-04-23 18:33:04.840258726 +0000 UTC m=+0.663112251 container remove 1f840779cda8cbf11e6a133c77bd8ae9dfddfa09119e0c009624b9d20678688f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:33:05.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T18:33:05.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:05.343 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T18:33:05.343 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T18:33:05.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.b 2024-04-23T18:33:05.668 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:05 smithi040 systemd[1]: Stopping Ceph mon.b for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:06.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-b[32620]: 2024-04-23T18:33:05.675+0000 7f35a7526700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:06.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-b[32620]: 2024-04-23T18:33:05.675+0000 7f35a7526700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:33:06.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:05 smithi040 podman[69457]: 2024-04-23 18:33:05.798455348 +0000 UTC m=+0.243996067 container died 34f7777fe01858d69829971aa925b562f3fc33410127d6f8005cc91b055ae7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-23T18:33:06.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:06 smithi040 podman[69457]: 2024-04-23 18:33:06.200714932 +0000 UTC m=+0.646255651 container cleanup 34f7777fe01858d69829971aa925b562f3fc33410127d6f8005cc91b055ae7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-23T18:33:06.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 23 18:33:06 smithi040 bash[69457]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-b 2024-04-23T18:33:06.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T18:33:06.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:06.615 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T18:33:06.615 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T18:33:06.615 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.c 2024-04-23T18:33:06.920 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:06 smithi064 systemd[1]: Stopping Ceph mon.c for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c[33057]: 2024-04-23T18:33:07.040+0000 7f964bd21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c[33057]: 2024-04-23T18:33:07.041+0000 7f964bd21700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:33:07.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:07 smithi064 podman[71800]: 2024-04-23 18:33:07.143654195 +0000 UTC m=+0.316002570 container died 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T18:33:07.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:07 smithi064 podman[71800]: 2024-04-23 18:33:07.543044731 +0000 UTC m=+0.715393103 container cleanup 70b853e25ec9fc0b08e475edeeb4aaca4b590927364ed4e3e4b46cc7d04b058f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T18:33:07.829 INFO:journalctl@ceph.mon.c.smithi064.stdout:Apr 23 18:33:07 smithi064 bash[71800]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mon-c 2024-04-23T18:33:07.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T18:33:08.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:08.026 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T18:33:08.026 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T18:33:08.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:33:08.335 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 systemd[1]: Stopping Ceph mgr.x for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:08.644 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:33:08.342+0000 7f45e0e92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:08.644 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x[29454]: 2024-04-23T18:33:08.342+0000 7f45e0e92700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-23T18:33:08.644 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 podman[96043]: 2024-04-23 18:33:08.374155069 +0000 UTC m=+0.145781133 container died d1cd4cf4d2da47d2e12633855369bbe517a36007c04acf06b99cca40fdfaab5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-04-23T18:33:09.006 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 podman[96043]: 2024-04-23 18:33:08.785543454 +0000 UTC m=+0.557169516 container cleanup d1cd4cf4d2da47d2e12633855369bbe517a36007c04acf06b99cca40fdfaab5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:33:09.006 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:08 smithi032 bash[96043]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x 2024-04-23T18:33:09.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T18:33:09.288 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:09 smithi032 podman[96057]: 2024-04-23 18:33:09.037265251 +0000 UTC m=+0.657588314 container remove d1cd4cf4d2da47d2e12633855369bbe517a36007c04acf06b99cca40fdfaab5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:33:09.288 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:09 smithi032 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-23T18:33:09.288 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:09 smithi032 systemd[1]: Stopped Ceph mgr.x for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:33:09.288 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 23 18:33:09 smithi032 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 39.152s CPU time. 2024-04-23T18:33:09.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:09.302 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T18:33:09.302 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T18:33:09.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.y 2024-04-23T18:33:09.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:09 smithi040 systemd[1]: Stopping Ceph mgr.y for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:10.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:09 smithi040 podman[69577]: 2024-04-23 18:33:09.635672311 +0000 UTC m=+0.122365777 container died 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:33:10.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:10 smithi040 podman[69577]: 2024-04-23 18:33:10.574032354 +0000 UTC m=+1.060725815 container cleanup 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-23T18:33:10.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:10 smithi040 bash[69577]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y 2024-04-23T18:33:11.331 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:11 smithi040 podman[69589]: 2024-04-23 18:33:11.068043684 +0000 UTC m=+1.428096596 container remove 24faf3f314052b77cf02494e3926e7d3ca23f5da63fbe20ea756dd68131ed329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-23T18:33:11.331 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 23 18:33:11 smithi040 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-23T18:33:11.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T18:33:11.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:11.389 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T18:33:11.389 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-23T18:33:11.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.z 2024-04-23T18:33:11.670 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:33:11 smithi064 systemd[1]: Stopping Ceph mgr.z for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:12.420 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:33:11 smithi064 podman[71922]: 2024-04-23 18:33:11.980587121 +0000 UTC m=+0.379584717 container died c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:33:13.288 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:33:13 smithi064 podman[71922]: 2024-04-23 18:33:13.045270166 +0000 UTC m=+1.444267756 container cleanup c5694f5f7bd5f2aea12f1dfbde71593ea71bdf902ab1aae4645282b630e08e4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-23T18:33:13.288 INFO:journalctl@ceph.mgr.z.smithi064.stdout:Apr 23 18:33:13 smithi064 bash[71922]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mgr-z 2024-04-23T18:33:13.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-23T18:33:13.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:13.472 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-23T18:33:13.472 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T18:33:13.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.0 2024-04-23T18:33:13.756 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:13 smithi032 systemd[1]: Stopping Ceph osd.0 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:14.256 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0[43724]: 2024-04-23T18:33:13.789+0000 7f4e200b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:14.256 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0[43724]: 2024-04-23T18:33:13.789+0000 7f4e200b9700 -1 osd.0 116 *** Got signal Terminated *** 2024-04-23T18:33:14.256 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0[43724]: 2024-04-23T18:33:13.789+0000 7f4e200b9700 -1 osd.0 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:19.600 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:19 smithi032 podman[96166]: 2024-04-23 18:33:19.343004841 +0000 UTC m=+5.671890760 container died bffc293312df5a3ce2e759a13d96ed2f63b47155d9116d51047bf5f199ed8029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:33:19.975 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:19 smithi032 podman[96166]: 2024-04-23 18:33:19.698913479 +0000 UTC m=+6.027799394 container cleanup bffc293312df5a3ce2e759a13d96ed2f63b47155d9116d51047bf5f199ed8029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:33:19.975 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:19 smithi032 bash[96166]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0 2024-04-23T18:33:20.256 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:19 smithi032 podman[96185]: 2024-04-23 18:33:19.983846159 +0000 UTC m=+0.639886332 container remove bffc293312df5a3ce2e759a13d96ed2f63b47155d9116d51047bf5f199ed8029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-23T18:33:20.756 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:20 smithi032 podman[96276]: 2024-04-23 18:33:20.222271003 +0000 UTC m=+0.018458596 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:33:20.756 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:20 smithi032 podman[96276]: 2024-04-23 18:33:20.326539665 +0000 UTC m=+0.122727257 container create c9a48dbced588b7582e5480ba583ded4a97fa68bc795b035f056565b51d5414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:33:21.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:20 smithi032 podman[96276]: 2024-04-23 18:33:20.9131519 +0000 UTC m=+0.709339495 container init c9a48dbced588b7582e5480ba583ded4a97fa68bc795b035f056565b51d5414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:33:21.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:20 smithi032 podman[96276]: 2024-04-23 18:33:20.917478065 +0000 UTC m=+0.713665658 container start c9a48dbced588b7582e5480ba583ded4a97fa68bc795b035f056565b51d5414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T18:33:21.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:21 smithi032 podman[96276]: 2024-04-23 18:33:21.012928874 +0000 UTC m=+0.809116469 container attach c9a48dbced588b7582e5480ba583ded4a97fa68bc795b035f056565b51d5414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:33:21.506 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 23 18:33:21 smithi032 podman[96276]: 2024-04-23 18:33:21.238858638 +0000 UTC m=+1.035046239 container died c9a48dbced588b7582e5480ba583ded4a97fa68bc795b035f056565b51d5414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:33:22.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T18:33:22.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:22.495 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T18:33:22.495 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T18:33:22.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.1 2024-04-23T18:33:22.805 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:22 smithi040 systemd[1]: Stopping Ceph osd.1 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:23.115 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1[37238]: 2024-04-23T18:33:22.812+0000 7fc4fe14f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:23.115 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1[37238]: 2024-04-23T18:33:22.812+0000 7fc4fe14f700 -1 osd.1 116 *** Got signal Terminated *** 2024-04-23T18:33:23.115 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1[37238]: 2024-04-23T18:33:22.812+0000 7fc4fe14f700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:28.797 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:28 smithi040 podman[69715]: 2024-04-23 18:33:28.396933324 +0000 UTC m=+5.691898333 container died 02eff54c2ffd6f7282b80a68a9924cd7190a89d9c3f887f528858c86429e7f99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-23T18:33:29.115 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:28 smithi040 podman[69715]: 2024-04-23 18:33:28.913347474 +0000 UTC m=+6.208312492 container cleanup 02eff54c2ffd6f7282b80a68a9924cd7190a89d9c3f887f528858c86429e7f99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:33:29.115 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:28 smithi040 bash[69715]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1 2024-04-23T18:33:29.557 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69732]: 2024-04-23 18:33:29.141413422 +0000 UTC m=+0.748631986 container remove 02eff54c2ffd6f7282b80a68a9924cd7190a89d9c3f887f528858c86429e7f99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T18:33:29.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69827]: 2024-04-23 18:33:29.566247061 +0000 UTC m=+0.112830192 container create 727b3b1c4cbae93f741b84bd26e7d8b80f0b0acd624566aec4bc4e8da24fd447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:33:29.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69827]: 2024-04-23 18:33:29.472086359 +0000 UTC m=+0.018669510 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:33:29.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69827]: 2024-04-23 18:33:29.811320207 +0000 UTC m=+0.357903339 container init 727b3b1c4cbae93f741b84bd26e7d8b80f0b0acd624566aec4bc4e8da24fd447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-23T18:33:29.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69827]: 2024-04-23 18:33:29.814508969 +0000 UTC m=+0.361092109 container start 727b3b1c4cbae93f741b84bd26e7d8b80f0b0acd624566aec4bc4e8da24fd447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T18:33:30.149 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:29 smithi040 podman[69827]: 2024-04-23 18:33:29.911949339 +0000 UTC m=+0.458532482 container attach 727b3b1c4cbae93f741b84bd26e7d8b80f0b0acd624566aec4bc4e8da24fd447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:33:30.449 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 23 18:33:30 smithi040 podman[69855]: 2024-04-23 18:33:30.185445058 +0000 UTC m=+0.021610155 container died 727b3b1c4cbae93f741b84bd26e7d8b80f0b0acd624566aec4bc4e8da24fd447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T18:33:30.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T18:33:30.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:30.955 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T18:33:30.956 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T18:33:30.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.2 2024-04-23T18:33:31.259 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:31 smithi064 systemd[1]: Stopping Ceph osd.2 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:31.670 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2[38185]: 2024-04-23T18:33:31.266+0000 7fa12f7dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:31.670 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2[38185]: 2024-04-23T18:33:31.266+0000 7fa12f7dd700 -1 osd.2 116 *** Got signal Terminated *** 2024-04-23T18:33:31.670 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2[38185]: 2024-04-23T18:33:31.266+0000 7fa12f7dd700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:37.118 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:36 smithi064 podman[72059]: 2024-04-23 18:33:36.820179737 +0000 UTC m=+5.655686394 container died da40e7fee159be9265c85895c14534cf8d9f9f9a7086e40d2cf9231612f36be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:33:37.420 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:37 smithi064 podman[72059]: 2024-04-23 18:33:37.209322224 +0000 UTC m=+6.044828881 container cleanup da40e7fee159be9265c85895c14534cf8d9f9f9a7086e40d2cf9231612f36be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:33:37.420 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:37 smithi064 bash[72059]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2 2024-04-23T18:33:37.822 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:37 smithi064 podman[72078]: 2024-04-23 18:33:37.494769904 +0000 UTC m=+0.679145043 container remove da40e7fee159be9265c85895c14534cf8d9f9f9a7086e40d2cf9231612f36be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:33:38.090 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:37 smithi064 podman[72173]: 2024-04-23 18:33:37.733860387 +0000 UTC m=+0.018810336 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:33:38.090 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:37 smithi064 podman[72173]: 2024-04-23 18:33:37.836656088 +0000 UTC m=+0.121606026 container create 46cb5bf742a33395d0cc891d7ec05d27c67afb803af6991df2f6f26fe84a3d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T18:33:38.401 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:38 smithi064 podman[72173]: 2024-04-23 18:33:38.097621405 +0000 UTC m=+0.382571343 container init 46cb5bf742a33395d0cc891d7ec05d27c67afb803af6991df2f6f26fe84a3d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T18:33:38.401 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:38 smithi064 podman[72173]: 2024-04-23 18:33:38.101455743 +0000 UTC m=+0.386405691 container start 46cb5bf742a33395d0cc891d7ec05d27c67afb803af6991df2f6f26fe84a3d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:33:38.401 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:38 smithi064 podman[72173]: 2024-04-23 18:33:38.197765834 +0000 UTC m=+0.482715784 container attach 46cb5bf742a33395d0cc891d7ec05d27c67afb803af6991df2f6f26fe84a3d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-23T18:33:38.670 INFO:journalctl@ceph.osd.2.smithi064.stdout:Apr 23 18:33:38 smithi064 podman[72173]: 2024-04-23 18:33:38.409068484 +0000 UTC m=+0.694018434 container died 46cb5bf742a33395d0cc891d7ec05d27c67afb803af6991df2f6f26fe84a3d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:33:39.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T18:33:39.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:39.053 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T18:33:39.053 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T18:33:39.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.3 2024-04-23T18:33:39.379 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:39 smithi032 systemd[1]: Stopping Ceph osd.3 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:39.696 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:39 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:39.386+0000 7f50b6b5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:39.696 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:39 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:39.386+0000 7f50b6b5d700 -1 osd.3 116 *** Got signal Terminated *** 2024-04-23T18:33:39.696 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:39 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:39.386+0000 7f50b6b5d700 -1 osd.3 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:40.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:39 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:39.704+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:39.702+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:40.436 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:40 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:40.168+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:40.756 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:40 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:40.443+0000 7f50b3151700 -1 osd.3 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.179435+0000 front 2024-04-23T18:33:14.179337+0000 (oldest deadline 2024-04-23T18:33:39.479551+0000) 2024-04-23T18:33:40.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:40 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:40.673+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:41.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:40.745+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:41.391 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:41 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:41.124+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:41.676 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:41 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:41.400+0000 7f50b3151700 -1 osd.3 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.179435+0000 front 2024-04-23T18:33:14.179337+0000 (oldest deadline 2024-04-23T18:33:39.479551+0000) 2024-04-23T18:33:42.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:41 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:41.684+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:42.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:41 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:41.627+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:42.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:41.753+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:42.404 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:42 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:42.131+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:42.669 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:42 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:42.412+0000 7f50b3151700 -1 osd.3 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.179435+0000 front 2024-04-23T18:33:14.179337+0000 (oldest deadline 2024-04-23T18:33:39.479551+0000) 2024-04-23T18:33:43.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:42 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:42.677+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:43.071 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:42.786+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:43.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:42 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:42.646+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:43.455 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:43 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:43.167+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:43.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:43 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:43.637+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:43.756 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:43 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3[50574]: 2024-04-23T18:33:43.462+0000 7f50b3151700 -1 osd.3 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.179435+0000 front 2024-04-23T18:33:14.179337+0000 (oldest deadline 2024-04-23T18:33:39.479551+0000) 2024-04-23T18:33:44.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:43 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:43.667+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:43.785+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:44.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:44 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:44.201+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:44.898 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:44 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:44.634+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:45.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:44 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:44.654+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:45.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:44.802+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:45.179 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:44 smithi032 podman[96396]: 2024-04-23 18:33:44.906620478 +0000 UTC m=+5.645856010 container died 7fe44ddf65b19d376f35cfe2e12540c3e24a6e7d00e2733ffed7cb39d7966f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T18:33:45.496 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:45 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:45.195+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:45.497 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:45 smithi032 podman[96396]: 2024-04-23 18:33:45.278904788 +0000 UTC m=+6.018140319 container cleanup 7fe44ddf65b19d376f35cfe2e12540c3e24a6e7d00e2733ffed7cb39d7966f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-23T18:33:45.497 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:45 smithi032 bash[96396]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3 2024-04-23T18:33:45.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:45.205+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:45.756 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:45 smithi032 podman[96415]: 2024-04-23 18:33:45.505149095 +0000 UTC m=+0.601479701 container remove 7fe44ddf65b19d376f35cfe2e12540c3e24a6e7d00e2733ffed7cb39d7966f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:33:45.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:45 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:45.650+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:46.045 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:45 smithi032 podman[96507]: 2024-04-23 18:33:45.704973596 +0000 UTC m=+0.018151779 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:33:46.045 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:45 smithi032 podman[96507]: 2024-04-23 18:33:45.822972374 +0000 UTC m=+0.136150535 container create 915f3cb3a517f3d23ad7454ecd91e039ca341498ef287e97b133a3a5f3ab9cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-23T18:33:46.045 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:46 smithi032 podman[96507]: 2024-04-23 18:33:46.050475759 +0000 UTC m=+0.363653925 container init 915f3cb3a517f3d23ad7454ecd91e039ca341498ef287e97b133a3a5f3ab9cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-23T18:33:46.045 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:46 smithi032 podman[96507]: 2024-04-23 18:33:46.053780006 +0000 UTC m=+0.366958162 container start 915f3cb3a517f3d23ad7454ecd91e039ca341498ef287e97b133a3a5f3ab9cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T18:33:46.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:45 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:45.651+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:46.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:45.838+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:46.351 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:46 smithi032 podman[96507]: 2024-04-23 18:33:46.150869214 +0000 UTC m=+0.464047386 container attach 915f3cb3a517f3d23ad7454ecd91e039ca341498ef287e97b133a3a5f3ab9cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:33:46.352 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:46 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:46.164+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:46.637 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 23 18:33:46 smithi032 podman[96507]: 2024-04-23 18:33:46.36073613 +0000 UTC m=+0.673914296 container died 915f3cb3a517f3d23ad7454ecd91e039ca341498ef287e97b133a3a5f3ab9cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T18:33:46.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:46.225+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:46.955 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:46 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:46.695+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:46.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T18:33:47.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:47.018 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T18:33:47.018 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T18:33:47.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.4 2024-04-23T18:33:47.045 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:46 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:46.666+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:47.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:46.822+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:47.256 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:47 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:47.132+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:47.344 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:47 smithi040 systemd[1]: Stopping Ceph osd.4 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:47.615 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:47 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:47.351+0000 7fe4f1492700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:47.615 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:47 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:47.351+0000 7fe4f1492700 -1 osd.4 116 *** Got signal Terminated *** 2024-04-23T18:33:47.615 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:47 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:47.351+0000 7fe4f1492700 -1 osd.4 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:47.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:47.231+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:48.005 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:47 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:47.722+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:48.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:47 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:47.676+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:48.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:47.778+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:48.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:48 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:48.144+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:48.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:48.277+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:49.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:48 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:48.712+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:49.048 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:48 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:48.682+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:49.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:48.736+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:49.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:49 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:49.055+0000 7fe4e9a7e700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:23.750505+0000 front 2024-04-23T18:33:23.750470+0000 (oldest deadline 2024-04-23T18:33:49.050640+0000) 2024-04-23T18:33:49.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:49.137+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:49.670 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:49 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:49.428+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:49.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:49 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:49.264+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:50.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:49.677+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:50.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:49 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:49.677+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:50.092 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:49 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:49.707+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:50.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:49 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:49.734+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:50.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:50 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:50.100+0000 7fe4e9a7e700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:23.750505+0000 front 2024-04-23T18:33:23.750470+0000 (oldest deadline 2024-04-23T18:33:49.050640+0000) 2024-04-23T18:33:50.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:50 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:50.099+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:50.670 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:50 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:50.472+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:50.671 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:50 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:50.228+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:50.671 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:50 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:50.228+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:51.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:50 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:50.675+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:51.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:50 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:50.675+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:51.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:50 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:50.671+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:51.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:50 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:50.772+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:51.479 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:51.234+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:51.480 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:51.234+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:51.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:51.149+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:51.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:51.149+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:51.615 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:51 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:51.134+0000 7fe4e9a7e700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:23.750505+0000 front 2024-04-23T18:33:23.750470+0000 (oldest deadline 2024-04-23T18:33:49.050640+0000) 2024-04-23T18:33:51.781 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:51.488+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:52.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:51.705+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:52.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:51 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:51.705+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:52.078 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:51 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:51.698+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:52.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:51.789+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:52.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:51 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:51.789+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:52.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:52 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4[43904]: 2024-04-23T18:33:52.085+0000 7fe4e9a7e700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:23.750505+0000 front 2024-04-23T18:33:23.750470+0000 (oldest deadline 2024-04-23T18:33:49.050640+0000) 2024-04-23T18:33:52.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:52.183+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:52.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:52.183+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:52.670 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:52 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:52.458+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:52.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:52 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:52.212+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:52.671 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:52 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:52.212+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:53.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:52.670+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:53.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:52.670+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:53.103 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:52 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:52.702+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:53.103 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:52 smithi040 podman[69935]: 2024-04-23 18:33:52.819631414 +0000 UTC m=+5.593169057 container died 8a96acc422d865bb1de4c2ba2d3b5487797575c291b8624a54d268fd8250148e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:33:53.104 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:52 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:52.714+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:53.104 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:52 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:52.714+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:53.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:52 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:52.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:53.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:52 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:52.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:53.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:53 smithi040 podman[69935]: 2024-04-23 18:33:53.202584108 +0000 UTC m=+5.976121757 container cleanup 8a96acc422d865bb1de4c2ba2d3b5487797575c291b8624a54d268fd8250148e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-23T18:33:53.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:53 smithi040 bash[69935]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4 2024-04-23T18:33:53.488 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:53.216+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:53.488 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:53.216+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:53.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:53.222+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:53.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:53.222+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:53.672 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:53 smithi040 podman[69954]: 2024-04-23 18:33:53.396210904 +0000 UTC m=+0.571785374 container remove 8a96acc422d865bb1de4c2ba2d3b5487797575c291b8624a54d268fd8250148e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:33:53.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:53.495+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:53.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:53.737+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:53.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:53 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:53.737+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:53.946 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:53 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:53.679+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:53.946 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:53 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:53.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:53.946 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:53 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:53.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:53.946 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:53 smithi040 podman[70045]: 2024-04-23 18:33:53.731700484 +0000 UTC m=+0.018601790 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:33:53.946 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:53 smithi040 podman[70045]: 2024-04-23 18:33:53.855493354 +0000 UTC m=+0.142394640 container create ee59643bf724f19096a3ec1071bf89f843b587ffb962553fa92d6c9051cc32d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:33:54.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:53.717+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:54.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:53 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:53.717+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:54.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:54 smithi040 podman[70045]: 2024-04-23 18:33:54.075188528 +0000 UTC m=+0.362089817 container init ee59643bf724f19096a3ec1071bf89f843b587ffb962553fa92d6c9051cc32d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:33:54.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:54 smithi040 podman[70045]: 2024-04-23 18:33:54.078359021 +0000 UTC m=+0.365260315 container start ee59643bf724f19096a3ec1071bf89f843b587ffb962553fa92d6c9051cc32d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-23T18:33:54.365 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:54 smithi040 podman[70045]: 2024-04-23 18:33:54.167091579 +0000 UTC m=+0.453992883 container attach ee59643bf724f19096a3ec1071bf89f843b587ffb962553fa92d6c9051cc32d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:33:54.485 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:54 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:54.240+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:54.485 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:54 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:54.240+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:54.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:54.261+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:54.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:54.261+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:54.670 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 23 18:33:54 smithi040 podman[70045]: 2024-04-23 18:33:54.383644691 +0000 UTC m=+0.670545981 container died ee59643bf724f19096a3ec1071bf89f843b587ffb962553fa92d6c9051cc32d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T18:33:54.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:54 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:54.492+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:54.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:54 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:54.698+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:54.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:54 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:54.698+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:54.979 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:54 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:54.678+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:54.979 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:54 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:54.714+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:54.979 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:54 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:54.714+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:55.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T18:33:55.005 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:54.692+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:55.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:54 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:54.692+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:55.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:33:55.079 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T18:33:55.079 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T18:33:55.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.5 2024-04-23T18:33:55.414 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 systemd[1]: Stopping Ceph osd.5 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:33:55.415 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:55.227+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:55.415 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:55.227+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:55.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:55.223+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:55.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:55.223+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:55.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:55.421+0000 7ff55a6b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:33:55.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:55.421+0000 7ff55a6b7700 -1 osd.5 116 *** Got signal Terminated *** 2024-04-23T18:33:55.670 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:55.421+0000 7ff55a6b7700 -1 osd.5 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:33:55.671 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:55.538+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:55.671 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:55.668+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:55.671 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:55 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:55.668+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:56.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:55.712+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:56.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:55 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:55.712+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:56.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:55 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:55.691+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:56.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:55 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:55.718+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:56.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:55 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:55.718+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:56.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:56 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:56.233+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:56.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:56 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:56.233+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:56.559 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:56 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:56.225+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:56.559 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:56 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:56.225+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:56.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:56 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:56.566+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:56.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:56 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:56.705+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:56.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:56 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:56.705+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:57.005 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:56 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:56.743+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:57.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:56 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:56.743+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:57.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:56 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:56.727+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:57.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:56 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:56.755+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:57.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:56 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:56.755+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:57.549 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:57.265+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:57.549 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:57.265+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:57.720 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:57 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:57.272+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:57.720 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:57 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:57.272+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:57.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:57.557+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:57.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:57.746+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:57.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:57.746+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:57.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:57 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:57.746+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:33:58.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:57 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:57.727+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:58.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:57 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:57.727+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:58.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:57 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:57.761+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:58.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:57 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:57.761+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:33:58.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:57 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:57.724+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:58.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:57 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:57.724+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:58.598 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:58.286+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:58.599 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:58.286+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:58.719 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:58.315+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:58.719 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:58.315+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:58.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:58.606+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:58.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:58.770+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:58.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:58.770+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:58.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:58 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:58.770+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:33:59.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:58.727+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:33:59.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:58.727+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:33:59.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:58.727+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:33:59.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:58 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:58.714+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:33:59.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:58 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:58.714+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:33:59.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:58 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:58.674+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:33:59.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:58 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:58.674+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:33:59.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:58 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:58.674+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:33:59.608 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:59.256+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:33:59.608 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:59.256+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:33:59.608 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:33:59.256+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.846629+0000 front 2024-04-23T18:33:35.846545+0000 (oldest deadline 2024-04-23T18:33:58.746758+0000) 2024-04-23T18:33:59.684 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:59 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:59.346+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:33:59.685 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:33:59 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:33:59.346+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:33:59.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:59.616+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:33:59.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:33:59.616+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:33:59.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:59.763+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:33:59.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:59.763+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:33:59.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:33:59 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:33:59.763+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:00.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:59 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:59.692+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:00.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:59 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:59.692+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:00.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:33:59 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:33:59.692+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:00.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:59 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:59.681+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:00.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:33:59 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:33:59.681+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:00.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:59 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:59.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:00.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:59 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:59.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:00.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:33:59 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:33:59.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:00.649 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:34:00.299+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.242611+0000 front 2024-04-23T18:33:17.242321+0000 (oldest deadline 2024-04-23T18:33:44.843201+0000) 2024-04-23T18:34:00.650 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:34:00.299+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.843967+0000 front 2024-04-23T18:33:24.844091+0000 (oldest deadline 2024-04-23T18:33:50.144012+0000) 2024-04-23T18:34:00.650 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5[45152]: 2024-04-23T18:34:00.299+0000 7ff552ca3700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.846629+0000 front 2024-04-23T18:33:35.846545+0000 (oldest deadline 2024-04-23T18:33:58.746758+0000) 2024-04-23T18:34:00.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:00.340+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:00.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:00.340+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:00.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:00.340+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:00.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:00.657+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:00.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:00.657+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:00.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:00.767+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:00.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:00.767+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:00.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:00 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:00.767+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:01.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:00.674+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:01.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:00.674+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:01.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:00 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:00.674+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:01.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:00.640+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:01.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:00.640+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:01.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:00.658+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:01.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:00.658+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:01.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:00 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:00.658+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:01.339 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:01 smithi064 podman[72292]: 2024-04-23 18:34:01.058463819 +0000 UTC m=+5.773477206 container died 64ca288e8d9c1b70b9110fec18c3c2f0739626a536b9caab291030b941151efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:34:01.659 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:01 smithi064 podman[72292]: 2024-04-23 18:34:01.447441691 +0000 UTC m=+6.162455080 container cleanup 64ca288e8d9c1b70b9110fec18c3c2f0739626a536b9caab291030b941151efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T18:34:01.659 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:01 smithi064 bash[72292]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5 2024-04-23T18:34:01.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:01.380+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:01.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:01.380+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:01.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:01.380+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:01.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:01 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:01.629+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:01.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:01 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:01.629+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:01.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:01 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:01.629+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:01.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:01 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:01.616+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:01.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:01 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:01.616+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:01.920 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:01 smithi064 podman[72309]: 2024-04-23 18:34:01.751706911 +0000 UTC m=+0.688622394 container remove 64ca288e8d9c1b70b9110fec18c3c2f0739626a536b9caab291030b941151efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-23T18:34:01.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:01 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:01.667+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:01.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:01 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:01.667+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:01.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:01 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:01.752+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:01.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:01 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:01.752+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:01.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:01 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:01.752+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:02.005 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:01.701+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:02.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:01.701+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:02.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:01 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:01.701+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:02.370 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:01.976336703 +0000 UTC m=+0.018521726 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:02.370 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:02.09183286 +0000 UTC m=+0.134017869 container create 07fe8da016298f38f82b110d37feba203c8b130f52989a48d422df8175ae98b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:34:02.633 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:02.378254895 +0000 UTC m=+0.420439911 container init 07fe8da016298f38f82b110d37feba203c8b130f52989a48d422df8175ae98b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-23T18:34:02.633 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:02.381992389 +0000 UTC m=+0.424177406 container start 07fe8da016298f38f82b110d37feba203c8b130f52989a48d422df8175ae98b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T18:34:02.633 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:02.486312953 +0000 UTC m=+0.528497973 container attach 07fe8da016298f38f82b110d37feba203c8b130f52989a48d422df8175ae98b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:02.672 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:02.349+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:02.672 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:02.349+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:02.672 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:02.349+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:02.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:02.650+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:02.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:02.650+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:02.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:02.602+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:02.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:02.602+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:02.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:02 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:02.602+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:02.920 INFO:journalctl@ceph.osd.5.smithi064.stdout:Apr 23 18:34:02 smithi064 podman[72400]: 2024-04-23 18:34:02.696770029 +0000 UTC m=+0.738955041 container died 07fe8da016298f38f82b110d37feba203c8b130f52989a48d422df8175ae98b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:34:02.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:02.640+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:02.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:02.640+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:02.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:02.703+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:02.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:02.703+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:02.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:02 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:02.703+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:03.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:02.679+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:03.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:02.679+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:03.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:02.679+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:03.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T18:34:03.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:03.375 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T18:34:03.375 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T18:34:03.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.6 2024-04-23T18:34:03.708 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 systemd[1]: Stopping Ceph osd.6 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:03.708 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:03.367+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:03.708 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:03.367+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:03.708 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:03.367+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:03.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:03.684+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:03.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:03.684+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:03.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:03.594+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:03.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:03.594+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:03.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:03 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:03.594+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:03.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:03.640+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:03.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:03.640+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:03.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:03.745+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:03.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:03.745+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:03.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:03 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:03.745+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.715+0000 7fa8ad7c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.715+0000 7fa8ad7c8700 -1 osd.6 116 *** Got signal Terminated *** 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.715+0000 7fa8ad7c8700 -1 osd.6 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.719+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.719+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:04.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:03 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:03.719+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:04.664 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:04.415+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:04.664 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:04.415+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:04.664 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:04.415+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:04.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:04.652+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:04.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:04.652+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:04.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:04.584+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:04.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:04.584+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:04.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:04 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:04.584+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:04.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:04.612+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:04.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:04.612+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:04.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:04.762+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:04.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:04.762+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:04.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:04 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:04.762+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:05.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:04.672+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:05.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:04.672+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:05.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:04 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:04.672+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:05.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:05.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:05.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:05.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:05.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:05.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:05.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:05.559+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:05.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:05.559+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:05.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:05.559+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:05.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:05.685+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:05.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:05.685+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:05.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:05 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:05.685+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:05.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:05.626+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:05.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:05.626+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:05.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:05.793+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:05.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:05.793+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:05.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:05 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:05.793+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:06.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:05.636+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:06.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:05.636+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:06.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:05 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:05.636+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:06.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:06.334+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:06.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:06.334+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:06.666 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:06.334+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:06.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:06.607+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:06.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:06.607+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:06.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:06.607+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:06.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:06.708+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:06.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:06.708+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:06.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:06 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:06.708+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:06.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:06.603+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:06.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:06.603+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:06.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:06.798+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:06.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:06.798+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:06.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:06 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:06.798+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:07.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:06.673+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:07.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:06.673+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:07.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:06.673+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:07.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:06 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:06.673+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.398476+0000 front 2024-04-23T18:33:42.398542+0000 (oldest deadline 2024-04-23T18:34:05.898579+0000) 2024-04-23T18:34:07.641 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:07.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:07.641 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:07.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:07.641 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:07.375+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:07.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:07.621+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:07.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:07.621+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:07.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:07.621+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:07.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:07.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:07.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:07.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:07.922 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:07.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:07.922 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:07 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:07.781+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:08.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:07.649+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:08.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:07.649+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:08.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:07.649+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:08.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:07.649+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.398476+0000 front 2024-04-23T18:33:42.398542+0000 (oldest deadline 2024-04-23T18:34:05.898579+0000) 2024-04-23T18:34:08.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:07.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:08.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:07.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:08.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:07.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:08.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:07.642+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:08.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:07.642+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:08.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:07.642+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:08.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:07 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:07.642+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:08.386+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:08.386+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:08.386+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:08.617+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.493443+0000 front 2024-04-23T18:33:17.493396+0000 (oldest deadline 2024-04-23T18:33:39.694246+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:08.617+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.095960+0000 front 2024-04-23T18:33:26.096181+0000 (oldest deadline 2024-04-23T18:33:48.995803+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:08.617+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.397307+0000 front 2024-04-23T18:33:35.397280+0000 (oldest deadline 2024-04-23T18:33:58.297487+0000) 2024-04-23T18:34:08.756 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:08 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6[57592]: 2024-04-23T18:34:08.617+0000 7fa8a5db4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.398476+0000 front 2024-04-23T18:33:42.398542+0000 (oldest deadline 2024-04-23T18:34:05.898579+0000) 2024-04-23T18:34:08.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:08.653+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:08.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:08.653+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:08.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:08.653+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:08.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:08.776+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:08.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:08.776+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:08.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:08.776+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:08.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:08 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:08.776+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:09.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:08.736+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:09.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:08.736+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:09.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:08.736+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:09.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:08.638+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:09.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:08.638+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:09.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:08.638+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:09.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:08 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:08.638+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:09.506 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:09 smithi032 podman[96627]: 2024-04-23 18:34:09.218329305 +0000 UTC m=+5.628925581 container died 6bd37cd0064c19eb7322563a74fe5987d0d3892b8bcec4d508ff84701cfdf0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T18:34:09.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:09.429+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:09.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:09.429+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:09.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:09.429+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:09.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:09 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:09.429+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:09.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:09.676+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:09.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:09.676+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:09.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:09.676+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:09.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:09.791+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:09.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:09.791+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:09.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:09.791+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:09.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:09 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:09.791+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:09.950 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:09 smithi032 podman[96627]: 2024-04-23 18:34:09.624288843 +0000 UTC m=+6.034885121 container cleanup 6bd37cd0064c19eb7322563a74fe5987d0d3892b8bcec4d508ff84701cfdf0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:09.950 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:09 smithi032 bash[96627]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6 2024-04-23T18:34:10.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:09.724+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:10.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:09.724+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:10.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:09.724+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:10.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:09.625+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:10.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:09.625+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:10.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:09.625+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:10.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:09 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:09.625+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:10.256 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:09 smithi032 podman[96644]: 2024-04-23 18:34:09.959036302 +0000 UTC m=+0.744537226 container remove 6bd37cd0064c19eb7322563a74fe5987d0d3892b8bcec4d508ff84701cfdf0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.212762425 +0000 UTC m=+0.018350443 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.318224716 +0000 UTC m=+0.123812707 container create c276a7ea251b27967302d48061fe3b088b8028d4d858a1954c225da8cc4b71b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.545964167 +0000 UTC m=+0.351552159 container init c276a7ea251b27967302d48061fe3b088b8028d4d858a1954c225da8cc4b71b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.549163016 +0000 UTC m=+0.354751011 container start c276a7ea251b27967302d48061fe3b088b8028d4d858a1954c225da8cc4b71b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.629305434 +0000 UTC m=+0.434893435 container attach c276a7ea251b27967302d48061fe3b088b8028d4d858a1954c225da8cc4b71b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:10.428+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:10.621 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:10.428+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:10.622 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:10.428+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:10.622 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:10 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:10.428+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:10.638+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:10.638+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:10.638+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:10.840+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:10.840+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:10.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:10.840+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:10.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:10 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:10.840+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:11.006 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 23 18:34:10 smithi032 podman[96734]: 2024-04-23 18:34:10.856009679 +0000 UTC m=+0.661597694 container died c276a7ea251b27967302d48061fe3b088b8028d4d858a1954c225da8cc4b71b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:34:11.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:10.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:11.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:10.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:11.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:10.690+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:11.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:10.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:11.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:10.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:11.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:10.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:11.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:10 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:10.639+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:11.574 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:11.416+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:11.575 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:11.416+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:11.575 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:11.416+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:11.575 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:11 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:11.416+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:11.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T18:34:11.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:11.646 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T18:34:11.646 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T18:34:11.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.7 2024-04-23T18:34:11.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:11.618+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:11.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:11.618+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:11.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:11.618+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:11.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:11.618+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:11.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.678+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:11.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.678+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:11.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.678+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:11.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 systemd[1]: Stopping Ceph osd.7 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:11.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:11.652+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:11.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:11.652+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:11.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:11.652+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:11.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:11.845+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:11.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:11.845+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:11.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:11.845+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:11.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:11 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:11.845+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:12.365 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.974+0000 7fcf62327700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:12.365 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.974+0000 7fcf62327700 -1 osd.7 116 *** Got signal Terminated *** 2024-04-23T18:34:12.365 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:11.974+0000 7fcf62327700 -1 osd.7 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:12.755 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:12.374+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:12.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:12.374+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:12.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:12.374+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:12.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:12 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:12.374+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:12.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:12.603+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:12.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:12.603+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:12.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:12.603+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:12.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:12.818+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:12.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:12.818+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:12.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:12.818+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:12.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:12 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:12.818+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:13.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:12.668+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:13.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:12.668+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:13.115 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:12.668+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:13.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:12.643+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:13.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:12.643+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:13.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:12.643+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:13.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:12 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:12.643+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:13.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:13.370+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:13.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:13.370+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:13.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:13.370+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:13.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:13.370+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:13.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:13 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:13.370+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:13.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:13.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:13.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:13.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:13.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:13.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:13.672+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:13.672+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:13.672+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:13.672+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:13.621+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:13.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:13.621+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:13.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:13.621+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:13.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:13.621+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:51.560337+0000 front 2024-04-23T18:33:51.560305+0000 (oldest deadline 2024-04-23T18:34:13.260461+0000) 2024-04-23T18:34:14.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:13.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:14.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:13.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:14.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:13.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:14.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:13 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:13.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:14.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:14.398+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:14.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:14.398+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:14.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:14.398+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:14.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:14.398+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:14.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:14 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:14.398+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:14.881 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:14.631+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:14.881 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:14.631+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:14.881 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:14.631+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:15.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:14.707+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:15.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:14.707+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:14.707+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:14.707+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:14.635+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:14.635+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:14.635+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:15.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:14 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:14.635+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:51.560337+0000 front 2024-04-23T18:33:51.560305+0000 (oldest deadline 2024-04-23T18:34:13.260461+0000) 2024-04-23T18:34:15.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:14.888+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:15.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:14.888+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:15.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:14.888+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:15.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:14.888+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:15.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:14 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:14.888+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:15.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:15.390+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:15.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:15.390+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:15.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:15.390+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:15.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:15.390+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:15.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:15 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:15.390+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:15.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:15.727+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:15.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:15.727+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:15.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:15.727+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:15.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:15.727+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:15.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:15.586+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:15.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:15.586+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:15.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:15.586+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:15.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:15 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:15.586+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:51.560337+0000 front 2024-04-23T18:33:51.560305+0000 (oldest deadline 2024-04-23T18:34:13.260461+0000) 2024-04-23T18:34:15.882 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:15.594+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:15.882 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:15.594+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:15.882 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:15.594+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:16.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:15.890+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:16.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:15.890+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:16.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:15.890+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:16.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:15.890+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:16.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:15 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:15.890+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:16.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:16 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:16.426+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:16.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:16 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:16.426+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:16.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:16 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:16.426+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:16.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:16 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:16.426+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:16.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:16 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:16.426+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:16.865 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:16.611+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.154113+0000 front 2024-04-23T18:33:26.154205+0000 (oldest deadline 2024-04-23T18:33:52.054330+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:16.611+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.356949+0000 front 2024-04-23T18:33:35.356792+0000 (oldest deadline 2024-04-23T18:33:57.656805+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:16.611+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.557713+0000 front 2024-04-23T18:33:43.557801+0000 (oldest deadline 2024-04-23T18:34:05.257812+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7[50740]: 2024-04-23T18:34:16.611+0000 7fcf5a913700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:51.560337+0000 front 2024-04-23T18:33:51.560305+0000 (oldest deadline 2024-04-23T18:34:13.260461+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:16.703+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:16.703+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:16.703+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:16.866 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:16 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:16.703+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:16.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:16.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:16.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:16.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:16.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:16.920 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:16.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:16.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:16.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:16.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:16 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:16.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:17.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:17 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:17.467+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:17.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:17 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:17.467+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:17.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:17 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:17.467+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:17.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:17 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:17.467+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:17.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:17 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:17.467+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:17.843 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:17.597+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:17.843 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:17.597+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:17.843 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:17.597+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:17.844 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:17.597+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:17 smithi040 podman[70150]: 2024-04-23 18:34:17.630619888 +0000 UTC m=+5.779221101 container died 18225a6d3dd4e6f84c53f59cf01f55eb7d1ddaac13d3368d3a16901e2f48c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:17.662+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:17.662+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:17.662+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:17.662+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:17.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:17.662+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:18.158 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:17 smithi040 podman[70150]: 2024-04-23 18:34:17.965772246 +0000 UTC m=+6.114373461 container cleanup 18225a6d3dd4e6f84c53f59cf01f55eb7d1ddaac13d3368d3a16901e2f48c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T18:34:18.158 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:17 smithi040 bash[70150]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7 2024-04-23T18:34:18.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:17.851+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:18.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:17.851+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:18.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:17.851+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:18.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:17.851+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:18.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:17 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:17.851+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:18.575 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70167]: 2024-04-23 18:34:18.166907533 +0000 UTC m=+0.537785319 container remove 18225a6d3dd4e6f84c53f59cf01f55eb7d1ddaac13d3368d3a16901e2f48c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:18.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:18 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:18.692+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:18.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:18 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:18.692+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:18.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:18 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:18.692+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:18.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:18 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:18.692+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:18.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:18 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:18.692+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:18.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70259]: 2024-04-23 18:34:18.58449508 +0000 UTC m=+0.113478515 container create d61b3a6fbc4c80bf1dd04395b8104293dbab90790535c5e0e72f234c1fca5f2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:34:18.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70259]: 2024-04-23 18:34:18.489694735 +0000 UTC m=+0.018678179 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:18.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70259]: 2024-04-23 18:34:18.812109233 +0000 UTC m=+0.341092662 container init d61b3a6fbc4c80bf1dd04395b8104293dbab90790535c5e0e72f234c1fca5f2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:34:18.866 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70259]: 2024-04-23 18:34:18.815301535 +0000 UTC m=+0.344284970 container start d61b3a6fbc4c80bf1dd04395b8104293dbab90790535c5e0e72f234c1fca5f2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:34:18.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:18.607+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:18.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:18.607+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:18.607+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:18.607+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:18.842+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:18.842+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:18.842+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:18.842+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:18.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:18 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:18.842+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:19.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:18 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:18.514+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:19.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:18 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:18.514+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:19.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:18 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:18.514+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:19.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:18 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:18.514+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:19.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:18 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:18.514+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:19.148 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:18 smithi040 podman[70259]: 2024-04-23 18:34:18.920600231 +0000 UTC m=+0.449583681 container attach d61b3a6fbc4c80bf1dd04395b8104293dbab90790535c5e0e72f234c1fca5f2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-23T18:34:19.148 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 23 18:34:19 smithi040 podman[70259]: 2024-04-23 18:34:19.123540096 +0000 UTC m=+0.652523527 container died d61b3a6fbc4c80bf1dd04395b8104293dbab90790535c5e0e72f234c1fca5f2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:34:19.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T18:34:19.780 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:19.780 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T18:34:19.780 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T18:34:19.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.8 2024-04-23T18:34:19.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:19.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:19.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:19.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:19.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:19.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:19.860 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:19.610+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:19.861 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:19 smithi064 systemd[1]: Stopping Ceph osd.8 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:19.907 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:19.560+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:19.907 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:19.560+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:19.907 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:19.560+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:19.907 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:19.560+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:19.907 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:19 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:19.560+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:20.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:19 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:19.740+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:20.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:19 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:19.740+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:20.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:19 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:19.740+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:20.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:19 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:19.740+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:20.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:19 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:19.740+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:20.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:19.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:20.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:19.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:20.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:19.867+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:20.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:19.868+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:20.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:19 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:19.868+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:20.636 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.260+0000 7f877adef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:20.636 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.260+0000 7f877adef700 -1 osd.8 116 *** Got signal Terminated *** 2024-04-23T18:34:20.636 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.260+0000 7f877adef700 -1 osd.8 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:20.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:20.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:20.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:20.644+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:20.857+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:20.857+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:20.857+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:20.857+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:20.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:20 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:20.857+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:21.005 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:20 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:20.568+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:21.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:20 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:20.568+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:21.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:20 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:20.568+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:21.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:20 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:20.568+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:21.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:20 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:20.568+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:21.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:20 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:20.762+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:21.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:20 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:20.762+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:21.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:20 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:20.762+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:21.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:20 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:20.762+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:21.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:20 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:20.762+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:21.920 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:21.617+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:21.617+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:21.617+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:21.617+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:21.850+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:21.850+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:21.850+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:21.850+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:21.921 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:21 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:21.850+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:22.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:21 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:21.582+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:22.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:21.742+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:22.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:21.742+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:22.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:21.742+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:22.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:21.742+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:22.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:21.742+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:22.867 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:22.574+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:22.867 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:22.574+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:22.867 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:22.574+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:22.867 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:22.574+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:22.867 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:22.574+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:56.731521+0000 front 2024-04-23T18:33:56.731432+0000 (oldest deadline 2024-04-23T18:34:22.031457+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:23.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:22 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:22.559+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:23.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:22 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:22.749+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:23.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:22 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:22.875+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:23.863 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:23.558+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:23.863 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:23.558+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:23.864 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:23.558+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:23.864 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:23.558+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:23.864 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:23.558+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:56.731521+0000 front 2024-04-23T18:33:56.731432+0000 (oldest deadline 2024-04-23T18:34:22.031457+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:24.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:23 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:23.538+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:24.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:23 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:23.759+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:24.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:24.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:24.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:24.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:24.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:24.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:23 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:23.871+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:24.909 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:24.586+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:24.725549+0000 front 2024-04-23T18:33:24.725602+0000 (oldest deadline 2024-04-23T18:33:49.425671+0000) 2024-04-23T18:34:24.909 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:24.586+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.827214+0000 front 2024-04-23T18:33:35.827257+0000 (oldest deadline 2024-04-23T18:33:58.727447+0000) 2024-04-23T18:34:24.909 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:24.586+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:42.228594+0000 front 2024-04-23T18:33:42.228669+0000 (oldest deadline 2024-04-23T18:34:06.829047+0000) 2024-04-23T18:34:24.909 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:24.586+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:47.930513+0000 front 2024-04-23T18:33:47.930533+0000 (oldest deadline 2024-04-23T18:34:16.731062+0000) 2024-04-23T18:34:24.909 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8[51721]: 2024-04-23T18:34:24.586+0000 7f87733db700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:56.731521+0000 front 2024-04-23T18:33:56.731432+0000 (oldest deadline 2024-04-23T18:34:22.031457+0000) 2024-04-23T18:34:25.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:25.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:25.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:25.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:25.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:25.007 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:24 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:24.531+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:25.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:25.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:25.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:25.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:25.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:25.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:24 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:24.775+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:25.163 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:24.917+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:26.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:25 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:25.524+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:26.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:26.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:26.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:26.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:26.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:26.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:25 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:25.817+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:26.119 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:25 smithi064 podman[72518]: 2024-04-23 18:34:25.830830989 +0000 UTC m=+5.841893291 container died cd4bb9ac4b6ad90f80c5f30b0c3a2721fcf7486cd55763376495213f446ca032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:34:26.119 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.904+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:26.119 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.904+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:26.120 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.904+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:26.120 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.904+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:26.120 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.905+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:26.120 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:25 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:25.905+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:26.420 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:26 smithi064 podman[72518]: 2024-04-23 18:34:26.311051056 +0000 UTC m=+6.322113403 container cleanup cd4bb9ac4b6ad90f80c5f30b0c3a2721fcf7486cd55763376495213f446ca032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-23T18:34:26.420 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:26 smithi064 bash[72518]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.488+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.488+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.488+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.488+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.488+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:26.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:26 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:26.489+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:26.878 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:26 smithi064 podman[72535]: 2024-04-23 18:34:26.629152089 +0000 UTC m=+0.793332456 container remove cd4bb9ac4b6ad90f80c5f30b0c3a2721fcf7486cd55763376495213f446ca032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:34:27.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:27.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:27.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:27.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:27.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:27.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:26 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:26.826+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:27.132 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:26 smithi064 podman[72622]: 2024-04-23 18:34:26.890695003 +0000 UTC m=+0.018959673 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:27.132 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:27 smithi064 podman[72622]: 2024-04-23 18:34:27.004920632 +0000 UTC m=+0.133185282 container create aa4e45340684dd41940215c7bec9058396801b4d40c37e26e0d4c1ed94d2bc79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:27.133 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:26 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:26.885+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:27.407 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:27 smithi064 podman[72622]: 2024-04-23 18:34:27.317734666 +0000 UTC m=+0.445999319 container init aa4e45340684dd41940215c7bec9058396801b4d40c37e26e0d4c1ed94d2bc79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:34:27.407 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:27 smithi064 podman[72622]: 2024-04-23 18:34:27.320979357 +0000 UTC m=+0.449244011 container start aa4e45340684dd41940215c7bec9058396801b4d40c37e26e0d4c1ed94d2bc79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:27.407 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:27 smithi064 podman[72622]: 2024-04-23 18:34:27.416147319 +0000 UTC m=+0.544411969 container attach aa4e45340684dd41940215c7bec9058396801b4d40c37e26e0d4c1ed94d2bc79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:34:27.670 INFO:journalctl@ceph.osd.8.smithi064.stdout:Apr 23 18:34:27 smithi064 podman[72622]: 2024-04-23 18:34:27.624085667 +0000 UTC m=+0.752350314 container died aa4e45340684dd41940215c7bec9058396801b4d40c37e26e0d4c1ed94d2bc79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:27.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:27 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:27.498+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:28.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:28.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:28.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:28.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:28.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:28.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:27 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:27.834+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:28.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.910+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:28.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.910+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:28.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.910+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:28.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.910+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:28.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.910+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:28.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:27 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:27.911+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:28.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T18:34:28.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:28.319 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T18:34:28.319 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T18:34:28.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.9 2024-04-23T18:34:28.627 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 systemd[1]: Stopping Ceph osd.9 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:28.628 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.458+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:29.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.635+0000 7fb1ffd4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:29.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.635+0000 7fb1ffd4c700 -1 osd.9 116 *** Got signal Terminated *** 2024-04-23T18:34:29.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:28 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:28.635+0000 7fb1ffd4c700 -1 osd.9 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:29.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.847+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:29.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.847+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:29.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.847+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:29.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.848+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:29.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.848+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:29.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:28 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:28.848+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:29.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:29.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:29.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:29.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:29.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:29.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:28 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:28.956+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:29.756 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:29 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:29.499+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:30.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:30.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:30.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:30.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:30.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:30.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:29 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:29.818+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:30.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:29.995+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:31.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:30 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:30.519+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:31.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:31.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:31.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:31.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:31.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:31.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:30 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:30.770+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:31.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:31.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:31.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:31.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:31.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:31.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:30 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:30.964+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:32.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:31 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:31.563+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:32.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:32.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:31 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:31.784+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:32.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:31 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:31.998+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:33.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:32 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:32.565+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:06.524437+0000 front 2024-04-23T18:34:06.524472+0000 (oldest deadline 2024-04-23T18:34:32.424484+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:33.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:33.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:32 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:32.772+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:33.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:33.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:33.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:33.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:33.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:33.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:32 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:32.971+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:34.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.715179+0000 front 2024-04-23T18:33:17.715209+0000 (oldest deadline 2024-04-23T18:33:40.015331+0000) 2024-04-23T18:34:34.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:26.918073+0000 front 2024-04-23T18:33:26.918040+0000 (oldest deadline 2024-04-23T18:33:50.418055+0000) 2024-04-23T18:34:34.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:35.119272+0000 front 2024-04-23T18:33:35.119380+0000 (oldest deadline 2024-04-23T18:33:59.819124+0000) 2024-04-23T18:34:34.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:43.920306+0000 front 2024-04-23T18:33:43.920326+0000 (oldest deadline 2024-04-23T18:34:08.620416+0000) 2024-04-23T18:34:34.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:48.621076+0000 front 2024-04-23T18:33:48.620992+0000 (oldest deadline 2024-04-23T18:34:13.221522+0000) 2024-04-23T18:34:34.007 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:34:00.223200+0000 front 2024-04-23T18:34:00.223265+0000 (oldest deadline 2024-04-23T18:34:21.223475+0000) 2024-04-23T18:34:34.007 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:33 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9[64462]: 2024-04-23T18:34:33.608+0000 7fb1fc340700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:06.524437+0000 front 2024-04-23T18:34:06.524472+0000 (oldest deadline 2024-04-23T18:34:32.424484+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:34.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:33 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:33.751+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:34.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:33.926+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:34.588 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:34 smithi032 podman[96850]: 2024-04-23 18:34:34.277495499 +0000 UTC m=+5.749276699 container died 8d42e77eded09ad36d5f6f07f4a34c5a9117e5396b41ccbac75db9831d3b5191 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T18:34:35.005 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:34 smithi032 podman[96850]: 2024-04-23 18:34:34.596428512 +0000 UTC m=+6.068209701 container cleanup 8d42e77eded09ad36d5f6f07f4a34c5a9117e5396b41ccbac75db9831d3b5191 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:34:35.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:34 smithi032 bash[96850]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9 2024-04-23T18:34:35.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:34 smithi032 podman[96867]: 2024-04-23 18:34:34.781088675 +0000 UTC m=+0.500135948 container remove 8d42e77eded09ad36d5f6f07f4a34c5a9117e5396b41ccbac75db9831d3b5191 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:35.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:35.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:34 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:34.720+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:35.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:34 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:34.906+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:35.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.215378575 +0000 UTC m=+0.106229298 container create 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:34:35.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.127364969 +0000 UTC m=+0.018215630 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:35.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.460411737 +0000 UTC m=+0.351262396 container init 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-23T18:34:35.506 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.463570725 +0000 UTC m=+0.354421384 container start 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:34:36.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.659421295 +0000 UTC m=+0.550271968 container attach 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-23T18:34:36.006 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:35 smithi032 podman[96956]: 2024-04-23 18:34:35.763681388 +0000 UTC m=+0.654532047 container died 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T18:34:36.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:36.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:36.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:36.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:36.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:36.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:36.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:35 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:35.738+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:36.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:36.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:36.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:36.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:36.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:36.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:35 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:35.886+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:36.320 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 23 18:34:36 smithi032 podman[96982]: 2024-04-23 18:34:36.313234357 +0000 UTC m=+0.541203514 container remove 8ee8267b9c08bb795d2f9492652f4d5c45bd70b759bb241f2727f0c1f21f32aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:36.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T18:34:36.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:36.366 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T18:34:36.366 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T18:34:36.366 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.10 2024-04-23T18:34:36.677 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 systemd[1]: Stopping Ceph osd.10 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:37.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.684+0000 7fb7b5d56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:37.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.684+0000 7fb7b5d56700 -1 osd.10 116 *** Got signal Terminated *** 2024-04-23T18:34:37.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.684+0000 7fb7b5d56700 -1 osd.10 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:37.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:37.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:37.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:37.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:37.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:37.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:37.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:36 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:36.779+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:37.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:36 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:36.879+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:38.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:38.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:38.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:38.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:38.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:38.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:38.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:37 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:37.800+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:38.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:37.855+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:39.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:39.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:39.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:39.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:39.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:39.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:39.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:38 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:38.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:39.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:39.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:39.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:39.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:39.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:39.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:39.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:38 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:38.861+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:40.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:40.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:40.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:40.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:40.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:40.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:40.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:39 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:39.794+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:40.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:40.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:40.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:39 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:39.899+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:17.281578+0000 front 2024-04-23T18:33:17.281559+0000 (oldest deadline 2024-04-23T18:33:41.281967+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:27.083583+0000 front 2024-04-23T18:33:27.083504+0000 (oldest deadline 2024-04-23T18:33:51.783708+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.784394+0000 front 2024-04-23T18:33:31.784412+0000 (oldest deadline 2024-04-23T18:33:58.184979+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.086309+0000 front 2024-04-23T18:33:41.086413+0000 (oldest deadline 2024-04-23T18:34:06.886672+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:52.187698+0000 front 2024-04-23T18:33:52.187679+0000 (oldest deadline 2024-04-23T18:34:16.788329+0000) 2024-04-23T18:34:41.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:59.590668+0000 front 2024-04-23T18:33:59.590494+0000 (oldest deadline 2024-04-23T18:34:22.490230+0000) 2024-04-23T18:34:41.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:40 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10[57600]: 2024-04-23T18:34:40.816+0000 7fb7b234a700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.32:6822 osd.6 since back 2024-04-23T18:34:08.092919+0000 front 2024-04-23T18:34:08.092877+0000 (oldest deadline 2024-04-23T18:34:30.992889+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:41.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:41.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:40 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:40.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:42.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:42.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:42.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:42.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:42.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:42.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:42.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:41 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:41.892+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:42.615 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:42 smithi040 podman[70377]: 2024-04-23 18:34:42.297802705 +0000 UTC m=+5.721981809 container died a2c255e16d55aa5b750e340a05017ce15d8e3b903bbb4c451d92730a3caca881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:34:42.994 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:42 smithi040 podman[70377]: 2024-04-23 18:34:42.751637708 +0000 UTC m=+6.175816803 container cleanup a2c255e16d55aa5b750e340a05017ce15d8e3b903bbb4c451d92730a3caca881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:34:42.994 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:42 smithi040 bash[70377]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10 2024-04-23T18:34:43.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:43.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:43.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:43.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:43.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:43.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:43.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:42 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:42.914+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:43.322 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70394]: 2024-04-23 18:34:43.002875338 +0000 UTC m=+0.699894039 container remove a2c255e16d55aa5b750e340a05017ce15d8e3b903bbb4c451d92730a3caca881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T18:34:43.581 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.233973412 +0000 UTC m=+0.018313967 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:43.581 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.337132045 +0000 UTC m=+0.121472627 container create 10695830e0d463bb16355c6f0e12be1ad5c51870f846add60d48fd487474aa21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:43.581 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.514525491 +0000 UTC m=+0.298866032 container init 10695830e0d463bb16355c6f0e12be1ad5c51870f846add60d48fd487474aa21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:34:43.581 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.517592172 +0000 UTC m=+0.301932718 container start 10695830e0d463bb16355c6f0e12be1ad5c51870f846add60d48fd487474aa21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T18:34:43.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.589359555 +0000 UTC m=+0.373700106 container attach 10695830e0d463bb16355c6f0e12be1ad5c51870f846add60d48fd487474aa21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T18:34:43.865 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 23 18:34:43 smithi040 podman[70488]: 2024-04-23 18:34:43.825204038 +0000 UTC m=+0.609544587 container died 10695830e0d463bb16355c6f0e12be1ad5c51870f846add60d48fd487474aa21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:34:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:44.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:44.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:44.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:43 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:43.919+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:44.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T18:34:44.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:44.457 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T18:34:44.457 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T18:34:44.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.11 2024-04-23T18:34:44.670 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 systemd[1]: Stopping Ceph osd.11 for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:45.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.789+0000 7fce38d3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.789+0000 7fce38d3f700 -1 osd.11 116 *** Got signal Terminated *** 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.789+0000 7fce38d3f700 -1 osd.11 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:45.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:44 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:44.929+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:46.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:46.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:46.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:46.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:46.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:46.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:46.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:45 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:45.967+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:47.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:47.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:47.420 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:47.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:47.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:47.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:47.421 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:46 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:46.931+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:48.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:48.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:48.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:48.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:48.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:48.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:48.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:47 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:47.887+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:49.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-23T18:33:14.674791+0000 front 2024-04-23T18:33:14.674814+0000 (oldest deadline 2024-04-23T18:33:39.374711+0000) 2024-04-23T18:34:49.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-23T18:33:25.776501+0000 front 2024-04-23T18:33:25.776532+0000 (oldest deadline 2024-04-23T18:33:51.676689+0000) 2024-04-23T18:34:49.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6804 osd.2 since back 2024-04-23T18:33:31.677220+0000 front 2024-04-23T18:33:31.677295+0000 (oldest deadline 2024-04-23T18:33:57.577365+0000) 2024-04-23T18:34:49.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-23T18:33:41.579329+0000 front 2024-04-23T18:33:41.579346+0000 (oldest deadline 2024-04-23T18:34:06.879288+0000) 2024-04-23T18:34:49.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-23T18:33:50.281591+0000 front 2024-04-23T18:33:50.281512+0000 (oldest deadline 2024-04-23T18:34:14.381389+0000) 2024-04-23T18:34:49.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.64:6812 osd.5 since back 2024-04-23T18:33:58.883348+0000 front 2024-04-23T18:33:58.883259+0000 (oldest deadline 2024-04-23T18:34:22.383272+0000) 2024-04-23T18:34:49.171 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:48 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11[58788]: 2024-04-23T18:34:48.924+0000 7fce3132b700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-23T18:34:13.886632+0000 front 2024-04-23T18:34:13.886536+0000 (oldest deadline 2024-04-23T18:34:38.586526+0000) 2024-04-23T18:34:50.670 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:50 smithi064 podman[72753]: 2024-04-23 18:34:50.417511609 +0000 UTC m=+5.765138554 container died 9c7cf827def9bbeff75f55a26f6bd0cd2326fd9bf8eadf9969725cd1c5b18342 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:34:50.940 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:50 smithi064 podman[72753]: 2024-04-23 18:34:50.756267567 +0000 UTC m=+6.103894520 container cleanup 9c7cf827def9bbeff75f55a26f6bd0cd2326fd9bf8eadf9969725cd1c5b18342 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T18:34:50.940 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:50 smithi064 bash[72753]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11 2024-04-23T18:34:51.376 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:50 smithi064 podman[72772]: 2024-04-23 18:34:50.949032973 +0000 UTC m=+0.525988580 container remove 9c7cf827def9bbeff75f55a26f6bd0cd2326fd9bf8eadf9969725cd1c5b18342 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:34:51.670 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.287892995 +0000 UTC m=+0.018263234 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:34:51.671 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.399968538 +0000 UTC m=+0.130338757 container create bdeee6ce332e53577bbd67ed928424fa4a4f8fdfabb47c20372e32c879df2fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-23T18:34:51.671 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.627262067 +0000 UTC m=+0.357632287 container init bdeee6ce332e53577bbd67ed928424fa4a4f8fdfabb47c20372e32c879df2fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T18:34:51.671 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.630462009 +0000 UTC m=+0.360832228 container start bdeee6ce332e53577bbd67ed928424fa4a4f8fdfabb47c20372e32c879df2fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T18:34:52.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.718818103 +0000 UTC m=+0.449188352 container attach bdeee6ce332e53577bbd67ed928424fa4a4f8fdfabb47c20372e32c879df2fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T18:34:52.170 INFO:journalctl@ceph.osd.11.smithi064.stdout:Apr 23 18:34:51 smithi064 podman[72862]: 2024-04-23 18:34:51.941518889 +0000 UTC m=+0.671889117 container died bdeee6ce332e53577bbd67ed928424fa4a4f8fdfabb47c20372e32c879df2fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:34:52.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T18:34:52.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:52.569 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T18:34:52.569 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-23T18:34:52.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.a 2024-04-23T18:34:52.874 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:52 smithi032 systemd[1]: Stopping Ceph mds.a for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:53.145 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a[71329]: 2024-04-23T18:34:52.881+0000 7f56e4528700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:53.145 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:52 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a[71329]: 2024-04-23T18:34:52.881+0000 7f56e4528700 -1 mds.a *** got signal Terminated *** 2024-04-23T18:34:57.206 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:56 smithi032 podman[97075]: 2024-04-23 18:34:56.891924028 +0000 UTC m=+4.107679579 container died 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-23T18:34:57.206 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:57 smithi032 podman[97075]: 2024-04-23 18:34:57.214940918 +0000 UTC m=+4.430696476 container cleanup 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:34:57.506 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:57 smithi032 bash[97075]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a 2024-04-23T18:34:57.506 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 23 18:34:57 smithi032 podman[97101]: 2024-04-23 18:34:57.391249997 +0000 UTC m=+0.496151194 container remove 519a8222111bb4d2a49e23b21e3c19f38efd46efac7b83dc3ece9e59279a8df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:34:57.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.a.service' 2024-04-23T18:34:57.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:34:57.678 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-23T18:34:57.678 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-23T18:34:57.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.d 2024-04-23T18:34:58.005 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:34:57 smithi032 systemd[1]: Stopping Ceph mds.d for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:34:58.505 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:34:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d[70327]: 2024-04-23T18:34:58.035+0000 7f9ef0627700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:34:58.506 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:34:58 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d[70327]: 2024-04-23T18:34:58.035+0000 7f9ef0627700 -1 mds.d *** got signal Terminated *** 2024-04-23T18:35:01.477 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:35:01 smithi032 podman[97207]: 2024-04-23 18:35:01.189818148 +0000 UTC m=+3.272965177 container died 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-23T18:35:01.756 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:35:01 smithi032 podman[97207]: 2024-04-23 18:35:01.560481742 +0000 UTC m=+3.643628766 container cleanup 2c566e5388ca502174da251fc52cf444a2bdfe5547aa3d9f7f21a55ae6b615fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:35:01.756 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 23 18:35:01 smithi032 bash[97207]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-d 2024-04-23T18:35:01.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.d.service' 2024-04-23T18:35:01.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:01.967 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-23T18:35:01.967 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-23T18:35:01.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.g 2024-04-23T18:35:02.255 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:02 smithi032 systemd[1]: Stopping Ceph mds.g for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:02.755 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g[70932]: 2024-04-23T18:35:02.322+0000 7fc9a2d85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:02.756 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:02 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g[70932]: 2024-04-23T18:35:02.322+0000 7fc9a2d85700 -1 mds.g *** got signal Terminated *** 2024-04-23T18:35:06.648 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:06 smithi032 podman[97328]: 2024-04-23 18:35:06.335750604 +0000 UTC m=+4.145056690 container died 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:35:06.952 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:06 smithi032 podman[97328]: 2024-04-23 18:35:06.656171879 +0000 UTC m=+4.465477961 container cleanup 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:35:06.952 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:06 smithi032 bash[97328]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g 2024-04-23T18:35:06.953 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 23 18:35:06 smithi032 podman[97342]: 2024-04-23 18:35:06.84927091 +0000 UTC m=+0.514157006 container remove 913fe3a9127dc35d507453e771d69e91570f72e0148185b796a70881b4f17962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T18:35:06.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.g.service' 2024-04-23T18:35:07.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:07.001 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-23T18:35:07.002 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-23T18:35:07.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.j 2024-04-23T18:35:07.256 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:07 smithi032 systemd[1]: Stopping Ceph mds.j for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:07.755 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j[69975]: 2024-04-23T18:35:07.360+0000 7f1d41dd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:07.755 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:07 smithi032 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j[69975]: 2024-04-23T18:35:07.360+0000 7f1d41dd8700 -1 mds.j *** got signal Terminated *** 2024-04-23T18:35:10.469 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 podman[97449]: 2024-04-23 18:35:10.224518343 +0000 UTC m=+2.994188994 container died 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:35:10.727 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 podman[97449]: 2024-04-23 18:35:10.551907861 +0000 UTC m=+3.321578521 container cleanup 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-23T18:35:10.727 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 bash[97449]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j 2024-04-23T18:35:10.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.j.service' 2024-04-23T18:35:10.994 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 podman[97463]: 2024-04-23 18:35:10.736197034 +0000 UTC m=+0.505137424 container remove 3df7c5d86506ba69a62711fe2ae806b4f9aa83874075ca4159ea57163ff6ff4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:35:10.994 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.j.service: Deactivated successfully. 2024-04-23T18:35:10.994 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 systemd[1]: Stopped Ceph mds.j for cdb712f8-019b-11ef-bc93-c7b262605968. 2024-04-23T18:35:10.994 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 23 18:35:10 smithi032 systemd[1]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.168s CPU time. 2024-04-23T18:35:11.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:11.004 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-23T18:35:11.004 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-23T18:35:11.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.b 2024-04-23T18:35:11.310 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:11 smithi040 systemd[1]: Stopping Ceph mds.b for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:11.586 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: 2024-04-23T18:35:11.318+0000 7f2b700f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:11.587 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:11 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b[62810]: 2024-04-23T18:35:11.318+0000 7f2b700f0700 -1 mds.b *** got signal Terminated *** 2024-04-23T18:35:12.659 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:12 smithi040 podman[70594]: 2024-04-23 18:35:12.406466392 +0000 UTC m=+1.191054126 container died d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:35:13.010 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:12 smithi040 podman[70594]: 2024-04-23 18:35:12.759806265 +0000 UTC m=+1.544394040 container cleanup d8be8c1e1b907a771ccace542c57c0fa33c76a3713731e34f7d16e72433ff285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:35:13.010 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 23 18:35:12 smithi040 bash[70594]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-b 2024-04-23T18:35:13.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.b.service' 2024-04-23T18:35:13.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:13.217 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-23T18:35:13.217 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-23T18:35:13.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.e 2024-04-23T18:35:13.563 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:13 smithi040 systemd[1]: Stopping Ceph mds.e for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:13.865 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e[62197]: 2024-04-23T18:35:13.572+0000 7f470f6c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:13.865 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:13 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e[62197]: 2024-04-23T18:35:13.572+0000 7f470f6c5700 -1 mds.e *** got signal Terminated *** 2024-04-23T18:35:16.688 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:16 smithi040 podman[70729]: 2024-04-23 18:35:16.404623027 +0000 UTC m=+2.948784837 container died 0cd2f5bd6c02ce412bfeceb6254567c0515d9108452c96a15ac445473b4e83c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:35:16.988 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:16 smithi040 podman[70729]: 2024-04-23 18:35:16.779742174 +0000 UTC m=+3.323903987 container cleanup 0cd2f5bd6c02ce412bfeceb6254567c0515d9108452c96a15ac445473b4e83c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5) 2024-04-23T18:35:16.989 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 23 18:35:16 smithi040 bash[70729]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-e 2024-04-23T18:35:17.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.e.service' 2024-04-23T18:35:17.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:17.163 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-23T18:35:17.163 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-23T18:35:17.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.h 2024-04-23T18:35:17.500 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:17 smithi040 systemd[1]: Stopping Ceph mds.h for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:17.865 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: 2024-04-23T18:35:17.509+0000 7f7bf8340700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:17.865 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:17 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h[62504]: 2024-04-23T18:35:17.509+0000 7f7bf8340700 -1 mds.h *** got signal Terminated *** 2024-04-23T18:35:20.699 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:20 smithi040 podman[70848]: 2024-04-23 18:35:20.402998875 +0000 UTC m=+3.008666887 container died e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T18:35:21.034 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:20 smithi040 podman[70848]: 2024-04-23 18:35:20.708603582 +0000 UTC m=+3.314271636 container cleanup e24e944140a46097dbaaf2c80448a97a63115fbd0adf3defe0c45b1d8996b922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T18:35:21.034 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 23 18:35:20 smithi040 bash[70848]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-h 2024-04-23T18:35:21.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.h.service' 2024-04-23T18:35:21.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:21.216 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-23T18:35:21.216 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-23T18:35:21.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.k 2024-04-23T18:35:21.615 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:21 smithi040 systemd[1]: Stopping Ceph mds.k for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:21.615 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k[61875]: 2024-04-23T18:35:21.537+0000 7ff2cdbff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:21.615 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:21 smithi040 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k[61875]: 2024-04-23T18:35:21.537+0000 7ff2cdbff700 -1 mds.k *** got signal Terminated *** 2024-04-23T18:35:23.615 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:23 smithi040 podman[70969]: 2024-04-23 18:35:23.323015586 +0000 UTC m=+1.881644763 container died 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:35:24.063 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:23 smithi040 podman[70969]: 2024-04-23 18:35:23.645503226 +0000 UTC m=+2.204132397 container cleanup 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:35:24.063 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:23 smithi040 bash[70969]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k 2024-04-23T18:35:24.063 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 23 18:35:23 smithi040 podman[70984]: 2024-04-23 18:35:23.854687632 +0000 UTC m=+0.525601071 container remove 6a550abe154a6b4534a74504ed4afcf442fc84c2e8595d79a650b2211056cbf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-23T18:35:24.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.k.service' 2024-04-23T18:35:24.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:24.110 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-23T18:35:24.110 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-23T18:35:24.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c 2024-04-23T18:35:24.402 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:24 smithi064 systemd[1]: Stopping Ceph mds.c for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:24.670 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: 2024-04-23T18:35:24.439+0000 7fb90bc29700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:24.670 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:24 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c[70428]: 2024-04-23T18:35:24.439+0000 7fb90bc29700 -1 mds.c *** got signal Terminated *** 2024-04-23T18:35:28.670 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:28 smithi064 podman[72966]: 2024-04-23 18:35:28.408935142 +0000 UTC m=+4.087192728 container died ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:35:29.032 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:28 smithi064 podman[72966]: 2024-04-23 18:35:28.797151123 +0000 UTC m=+4.475408715 container cleanup ff8cccc70ed41085f06579887c744c113254574a4a5e95ef7f0688598b67ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:35:29.032 INFO:journalctl@ceph.mds.c.smithi064.stdout:Apr 23 18:35:28 smithi064 bash[72966]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-c 2024-04-23T18:35:29.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.c.service' 2024-04-23T18:35:29.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:29.217 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-23T18:35:29.217 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-23T18:35:29.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.f 2024-04-23T18:35:29.670 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:29 smithi064 systemd[1]: Stopping Ceph mds.f for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:29.670 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: 2024-04-23T18:35:29.559+0000 7f0d472cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:29.670 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:29 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f[63932]: 2024-04-23T18:35:29.559+0000 7f0d472cc700 -1 mds.f *** got signal Terminated *** 2024-04-23T18:35:32.670 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:32 smithi064 podman[73099]: 2024-04-23 18:35:32.405726006 +0000 UTC m=+2.952634891 container died 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:35:33.126 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:32 smithi064 podman[73099]: 2024-04-23 18:35:32.88419012 +0000 UTC m=+3.431099010 container cleanup 42a2f0b8869d769bdd998e84d6839469371edf76140641018b4255d125c3f41a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:35:33.126 INFO:journalctl@ceph.mds.f.smithi064.stdout:Apr 23 18:35:32 smithi064 bash[73099]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-f 2024-04-23T18:35:33.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.f.service' 2024-04-23T18:35:33.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:33.286 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-23T18:35:33.286 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-23T18:35:33.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.i 2024-04-23T18:35:33.420 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:33 smithi064 systemd[1]: Stopping Ceph mds.i for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:33.920 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: 2024-04-23T18:35:33.612+0000 7f677cfd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:33.920 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:33 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i[64553]: 2024-04-23T18:35:33.612+0000 7f677cfd8700 -1 mds.i *** got signal Terminated *** 2024-04-23T18:35:36.646 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:36 smithi064 podman[73220]: 2024-04-23 18:35:36.403355989 +0000 UTC m=+2.888951456 container died 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416) 2024-04-23T18:35:36.920 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:36 smithi064 podman[73220]: 2024-04-23 18:35:36.75438438 +0000 UTC m=+3.239979897 container cleanup 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:35:36.920 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:36 smithi064 bash[73220]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i 2024-04-23T18:35:37.368 INFO:journalctl@ceph.mds.i.smithi064.stdout:Apr 23 18:35:37 smithi064 podman[73234]: 2024-04-23 18:35:37.084009323 +0000 UTC m=+0.675733568 container remove 6db7a1fe6ce211aa6a7d31ecb938827c12c5b26017118df78750bac62ac8af7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:35:37.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.i.service' 2024-04-23T18:35:37.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:37.415 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-23T18:35:37.415 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-23T18:35:37.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.l 2024-04-23T18:35:37.670 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:37 smithi064 systemd[1]: Stopping Ceph mds.l for cdb712f8-019b-11ef-bc93-c7b262605968... 2024-04-23T18:35:38.170 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l[63619]: 2024-04-23T18:35:37.756+0000 7fd768c24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:35:38.170 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:37 smithi064 ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l[63619]: 2024-04-23T18:35:37.756+0000 7fd768c24700 -1 mds.l *** got signal Terminated *** 2024-04-23T18:35:40.400 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:40 smithi064 podman[73342]: 2024-04-23 18:35:40.149293167 +0000 UTC m=+2.506086476 container died 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:35:40.670 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:40 smithi064 podman[73342]: 2024-04-23 18:35:40.616468602 +0000 UTC m=+2.973261919 container cleanup 02798a04ebcaaf18c4b22855a25ac4fa868310e39515652489ce7b36d75460bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:35:40.670 INFO:journalctl@ceph.mds.l.smithi064.stdout:Apr 23 18:35:40 smithi064 bash[73342]: ceph-cdb712f8-019b-11ef-bc93-c7b262605968-mds-l 2024-04-23T18:35:40.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cdb712f8-019b-11ef-bc93-c7b262605968@mds.l.service' 2024-04-23T18:35:41.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:35:41.002 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-23T18:35:41.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:35:41.203 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:35:43.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:35:43.942 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:35:46.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:35:46.726 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:35:49.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:35:49.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:35:49.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:35:49.315 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T18:35:49.317 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi032/crash 2024-04-23T18:35:49.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:35:49.361 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:35:49.361 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:35:49.363 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi040/crash 2024-04-23T18:35:49.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:35:49.400 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:35:49.400 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:35:49.402 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi064/crash 2024-04-23T18:35:49.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:35:49.447 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:35:49.448 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:35:49.448 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T18:35:49.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-23T18:35:49.490 INFO:tasks.cephadm:Compressing logs... 2024-04-23T18:35:49.491 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:35:49.534 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:35:49.536 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:35:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T18:35:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:35:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:35:49.573 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:35:49.573 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-04-23T18:35:49.573 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.5.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.0% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5%/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.i.log: -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:35:49.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T18:35:49.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:35:49.575 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T18:35:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.5% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-23T18:35:49.580 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-23T18:35:49.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-23T18:35:49.582 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:35:49.583 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:: No such file or directory 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.7.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.h.log: 87.0% 90.5% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:35:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T18:35:49.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.9.log: 91.1% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:35:49.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-23T18:35:49.596 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-23T18:35:49.598 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:35:49.606 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-23T18:35:49.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-23T18:35:49.621 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:35:49.803 INFO:teuthology.orchestra.run.smithi064.stderr: 93.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-23T18:35:49.926 INFO:teuthology.orchestra.run.smithi040.stderr: 93.9% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-23T18:35:50.033 INFO:teuthology.orchestra.run.smithi032.stderr: 90.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T18:35:50.042 INFO:teuthology.orchestra.run.smithi064.stderr: 91.9% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T18:35:50.153 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-23T18:35:50.212 INFO:teuthology.orchestra.run.smithi040.stderr: 92.0% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T18:35:50.783 INFO:teuthology.orchestra.run.smithi032.stderr: 90.2% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T18:35:52.667 INFO:teuthology.orchestra.run.smithi064.stderr: 95.7% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-23T18:35:53.695 INFO:teuthology.orchestra.run.smithi040.stderr: 94.1% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-23T18:35:54.005 INFO:teuthology.orchestra.run.smithi040.stderr: 93.9% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-23T18:36:16.950 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T18:36:25.294 INFO:teuthology.orchestra.run.smithi064.stderr: 91.5% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T18:36:28.450 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T18:36:30.409 INFO:teuthology.orchestra.run.smithi040.stderr: 91.6% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T18:36:34.264 INFO:teuthology.orchestra.run.smithi040.stderr: 91.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T18:36:35.465 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T18:36:37.698 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T18:36:37.912 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T18:36:37.914 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-23T18:36:37.915 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m48.364s 2024-04-23T18:36:37.915 INFO:teuthology.orchestra.run.smithi064.stderr:user 2m57.056s 2024-04-23T18:36:37.915 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m5.094s 2024-04-23T18:36:41.107 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T18:36:41.299 INFO:teuthology.orchestra.run.smithi040.stderr: 91.4% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T18:36:41.302 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-23T18:36:41.302 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m51.750s 2024-04-23T18:36:41.302 INFO:teuthology.orchestra.run.smithi040.stderr:user 3m12.547s 2024-04-23T18:36:41.302 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m5.430s 2024-04-23T18:36:43.446 INFO:teuthology.orchestra.run.smithi032.stderr: 91.1% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T18:36:50.299 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/cdb712f8-019b-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T18:36:50.301 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-23T18:36:50.301 INFO:teuthology.orchestra.run.smithi032.stderr:real 1m0.752s 2024-04-23T18:36:50.301 INFO:teuthology.orchestra.run.smithi032.stderr:user 2m57.344s 2024-04-23T18:36:50.301 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m4.935s 2024-04-23T18:36:50.301 INFO:tasks.cephadm:Archiving logs... 2024-04-23T18:36:50.302 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi032/log 2024-04-23T18:36:50.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:37:21.512 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi040/log 2024-04-23T18:37:21.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:37:53.649 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi064/log 2024-04-23T18:37:53.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:38:24.921 INFO:tasks.cephadm:Removing cluster... 2024-04-23T18:38:24.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force 2024-04-23T18:38:25.125 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:38:25.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force 2024-04-23T18:38:25.866 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:38:26.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cdb712f8-019b-11ef-bc93-c7b262605968 --force 2024-04-23T18:38:26.604 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: cdb712f8-019b-11ef-bc93-c7b262605968 2024-04-23T18:38:27.123 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T18:38:27.123 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:38:27.143 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:38:27.162 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:38:27.179 INFO:tasks.cephadm:Teardown complete 2024-04-23T18:38:27.179 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T18:38:27.192 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T18:38:27.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:38:27.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:38:27.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:38:27.312 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T18:38:27.330 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T18:38:27.330 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:38:27.334 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:38:27.337 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:38:27.348 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:38:27.351 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:38:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:^- hv01.front.sepia.ceph.com 2 6 377 53 +815us[ +815us] +/- 596ms 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -283us[ -255us] +/- 72ms 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +349us[ +376us] +/- 86ms 2024-04-23T18:38:27.354 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:38:27.356 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:38:27.356 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-23T18:38:27.356 INFO:teuthology.orchestra.run.smithi064.stdout:^- hv01.front.sepia.ceph.com 2 6 377 57 +814us[ +837us] +/- 596ms 2024-04-23T18:38:27.356 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -285us[ -262us] +/- 72ms 2024-04-23T18:38:27.357 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +355us[ +378us] +/- 86ms 2024-04-23T18:38:27.357 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:^- hv01.front.sepia.ceph.com 2 6 377 53 +829us[ +829us] +/- 596ms 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -277us[ -249us] +/- 72ms 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +352us[ +352us] +/- 86ms 2024-04-23T18:38:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:38:27.359 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T18:38:27.371 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T18:38:27.371 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T18:38:27.476 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:38:27.560 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:38:27.633 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:38:27.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:38:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:27.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:38:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:27.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:38:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:27.791 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T18:38:27.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T18:38:27.825 INFO:teuthology.task.internal:Duration was 2508.102322 seconds 2024-04-23T18:38:27.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T18:38:27.843 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T18:38:27.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:38:27.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:38:27.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:38:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:38:27.886 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:38:27.887 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:38:28.409 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T18:38:28.409 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-23T18:38:28.409 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:38:28.434 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-23T18:38:28.434 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:38:28.473 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-04-23T18:38:28.473 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:38:28.505 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T18:38:28.506 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:38:28.508 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:38:28.515 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:38:28.572 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T18:38:28.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:38:28.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:38:28.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:38:28.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T18:38:28.990 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T18:38:28.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:38:29.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:38:29.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:38:29.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T18:38:29.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.102 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-23T18:38:29.104 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-23T18:38:29.127 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-04-23T18:38:29.148 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:29.169 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:29.184 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:38:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:38:29.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T18:38:29.218 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T18:38:29.219 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi032 2024-04-23T18:38:29.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:38:29.261 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi040 2024-04-23T18:38:29.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:38:29.297 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091/remote/smithi064 2024-04-23T18:38:29.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:38:29.333 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T18:38:29.333 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:38:29.335 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:38:29.338 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:38:29.382 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T18:38:29.397 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T18:38:29.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T18:38:29.414 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T18:38:29.414 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:38:29.417 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:38:29.419 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:38:29.429 INFO:teuthology.orchestra.run.smithi032.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:38 /home/ubuntu/cephtest 2024-04-23T18:38:29.431 INFO:teuthology.orchestra.run.smithi040.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:38 /home/ubuntu/cephtest 2024-04-23T18:38:29.437 INFO:teuthology.orchestra.run.smithi064.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:38 /home/ubuntu/cephtest 2024-04-23T18:38:29.438 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T18:38:29.489 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T18:38:29.575 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T18:38:29.604 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-23T18:38:29.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:38:29.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.353418', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLStkaM79fMI2aiYOFzfuOrR7Pq5GZtlS9Df+QqNdgyw0pmUlEUf7LwEw6yIuoHeD/tX4MYrJ09Pif9wFHUJgXQ='} 2024-04-23T18:38:29.635 DEBUG:teuthology.nuke:shortname: smithi040 2024-04-23T18:38:29.635 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:38:29.659 DEBUG:teuthology.nuke:shortname: smithi064 2024-04-23T18:38:29.659 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:38:29.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.351380', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMiGL3h/cBsdICJfyfAwFkBowhz33uEV0Dy+8E6s9MPLwoYkM4k/+/l4dMnrMkScPGYSGCrJ8By3qFroDIEzct4='} 2024-04-23T18:38:29.684 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:50:37.355171', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAkVgZ4naHz37gpAwJEFiEu8ngNYQc1NzoQpuVbbXr4+aVIvC6jXHEerQMAiU/Zn8grtVSSqYF3BCYdcNFJnlGQ='} 2024-04-23T18:38:29.702 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-23T18:38:29.703 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:38:29.741 INFO:teuthology.orchestra.console.smithi040:Power off 2024-04-23T18:38:29.742 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:38:29.755 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-23T18:38:29.755 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:29.781 INFO:teuthology.orchestra.console.smithi064:Power off 2024-04-23T18:38:29.783 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:38:29.796 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-04-23T18:38:29.797 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:29.804 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-23T18:38:29.823 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-04-23T18:38:29.824 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:29.836 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-23T18:38:29.856 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-23T18:38:33.805 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:33.837 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:33.857 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:33.917 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-23T18:38:33.948 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-23T18:38:33.968 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-23T18:38:37.923 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:37.950 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:37.970 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:38:38.033 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-23T18:38:38.065 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-04-23T18:38:38.079 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-04-23T18:38:38.135 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-23T18:38:38.166 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-04-23T18:38:38.180 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-04-23T18:38:38.285 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 2508.102321624756 failure_reason: 'error during quiesce thrashing: Error releasing set ''23f38b46'': 1 (EPERM)' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-23T18:38:38.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T18:38:38.373 INFO:teuthology.run:FAIL